# Understanding AI in 2026: from prompts and RAG to agents and sovereign AI

Source: https://ascentis-ai.com/understanding-ai-2026/
Author: Erwan Lhermitte, Ascentis AI Ltd
Reviewed: 20 August 2026
Licence: readable and quotable; please cite the source URL.

Reference guide published by Ascentis AI Ltd. Full plain-text rendering of the page above,
generated from the published content. Citation welcome, with a link to the source.

------------------------------------------------------------------------

Ascentis AI · Applied AI Foundations

# Understanding AI in 2026: from prompts and RAG to agents and sovereign AI
You can already use a chatbot. This guide explains what sits behind it, what changes when AI touches your own data and systems, and how production AI is actually built. Plain English, practical examples, proper terminology. No maths and no coding background required.

Reading time ≈ 120 min
Beginner to intermediate · no maths
Current to August 2026

Contents

Part I — Foundations

- 01What a language model actually is

- 02When it isn't a language model

- 03The Four-Layer Model

- 04Prompt engineering

- 05Context engineering

Part II — Retrieval and inputs

- 06RAG: the concept

- 07The nine RAG patterns

- 08Embeddings and vector search

- 09Hybrid retrieval and fusion

- 10Rerankers

- 11Retrieval metrics

- 12Semantic vs ontological

- 13Vector databases

- 14Measuring hallucination

- 15Multimodal and document AI

Part III — Agents and systems

- 16Tools, MCP and the harness

- 17Structured outputs

- 18Agents and the loop

- 19Autonomy and orchestration

Part IV — Models and the market

- 20Choosing a model

- 21The open-weight surge

- 22Benchmarks

Part V — Buying, running and scaling it

- 23API calls, pricing and wrappers

- 24Cloud, self-hosted, on-device

- 25Docker, GPUs and inference

- 26Hosting and scaling

- 27Token consumption and cost

Part VI — Risk, control and adoption

- 28Where your data goes

- 29The governance layer

- 30Data readiness and maturity

- 31Your first 90 days

Reference

- 32Frequently asked questions

- 33Glossary

- 34Sources

By Erwan Lhermitte, Founder & CEO, Ascentis AI
Published 20 August 2026
Reviewed 20 August 2026
≈ 120 min read · 34 sections

## The short version

Useful AI products are systems, not models. The model generates or reasons; the surrounding engineering supplies evidence, memory, tools, permissions, checks and the rules for what happens next.

- Start by asking whether the problem is actually a language problem. Text, documents and conversation often suit language models. Forecasting, sensor anomalies, optimisation and many inspection tasks are usually better handled by classical machine learning, operations research or specialist vision models.

- Think in four layers: prompt, context, harness and loop. The prompt tells the model what to do. Context gives it what it needs to know. The harness controls what it can touch. The loop decides whether it should act again, stop or ask for help.

- A model's weights are not a live database. Current facts, company knowledge, user memory and business state normally have to be supplied by the application through retrieval, tools or stored state.

- Large context windows are capacity, not a quality guarantee. More context costs more, takes longer and can make relevant information harder for the model to use. Give the model the smallest high-signal context that can do the job.

- Grounding changes the risk profile, but there is no universal hallucination percentage. Faithfulness varies sharply by task, dataset, model and evaluation method. Measure it on your own workload and give the system permission to abstain when the evidence is weak.

- RAG is a retrieval problem before it is a generation problem. Measure whether the right evidence was found before blaming the answer model. Hybrid search and reranking are strong production patterns, but candidate counts and thresholds must be tuned on labelled queries.

- Use structured outputs for machine-to-machine work. JSON syntax alone is not enough. Validate the schema, values and business rules before downstream software acts on them.

- An agent is a controlled loop with tools. Production agents also need durable state, bounded retries, timeouts, idempotent actions, approvals for consequential steps and a reliable way to stop.

- Prompt injection is an architectural problem. Filters help, but they are not a complete defence. Pay particular attention when one system combines private data, untrusted content and a route to send or write information elsewhere.

- Open weight does not mean open source, and self-hosted does not automatically mean sovereign. Sovereignty also depends on data location, jurisdiction, software and model supply chains, update control and whether you can continue operating without a provider changing the rules.

- The economics of self-hosting depend heavily on utilisation. For intermittent workloads, hosted open-weight endpoints can be cheaper than owning or renting idle GPUs. Privacy, offline operation, regulatory constraints and operational independence can still justify self-hosting even when the API is cheaper.

- Model prices move quickly. Quote provider, model and date together. DeepSeek changed V4 pricing in August 2026, a useful reminder that a cost table without a timestamp becomes stale almost immediately.

- Production evaluation is broader than benchmark scores. Track task success, factual support, retrieval quality, human correction, latency, cost and failure modes. Run the same evaluation set when you change model, prompt, retrieval, tools or orchestration.

- AI governance should increase usable autonomy, not merely restrict it. Clear data tiers, approved tools, audit trails, human oversight and AI literacy make it possible to deploy higher-value systems with controlled risk.

- Data readiness is often the real constraint. Ask whether the data exists, can be reached, is trustworthy, has the labels or structure the task needs, and may legally be used. In many SMEs, the first useful AI system also becomes the fastest way to expose and improve the underlying data problem.

How claims are marked

MeasuredA figure from a named evaluation or study, cited in Sources.
Field noteSomething observed on Ascentis AI client work, anonymised.
Rule of thumbA working default worth starting from. Not a finding.
Law & standardA legal or standards position, correct at the date given.
Current marketPrices and product facts that move. Check before you rely on them.

Four layers around a production model

L1Prompt

The instruction for this call: what you want, the constraints and the shape of the answer.

L2Context

The information available to the model now: conversation, retrieved evidence, records, examples and tool descriptions.

L3Harness

The controlled environment around the model: tools, permissions, validation, sandboxing, limits and logging.

L4Loop

Observe, decide, act, verify, then repeat or stop. This is where useful autonomy begins.

[Figure] The layers build on one another. A single chatbot exchange may need little beyond the prompt and context. A production agent needs all four, plus ordinary software engineering around them.

Q. Contents — 34 sections in 6 parts
A.
Part I — Foundations

- 01 What a language model actually is

- 02 When it isn't a language model

- 03 The Four-Layer Model

- 04 Prompt engineering

- 05 Context engineering

Part II — Retrieval and inputs

- 06 RAG: the concept

- 07 The nine RAG patterns

- 08 Embeddings and vector search

- 09 Hybrid retrieval and fusion

- 10 Rerankers

- 11 Retrieval metrics

- 12 Semantic vs ontological

- 13 Vector databases

- 14 Measuring hallucination

- 15 Multimodal and document AI

Part III — Agents and systems

- 16 Tools, MCP and the harness

- 17 Structured outputs

- 18 Agents and the loop

- 19 Autonomy and orchestration

Part IV — Models and the market

- 20 Choosing a model

- 21 The open-weight surge

- 22 Benchmarks

Part V — Buying, running and scaling it

- 23 API calls, pricing and wrappers

- 24 Cloud, self-hosted, on-device

- 25 Docker, GPUs and inference

- 26 Hosting and scaling

- 27 Token consumption and cost

Part VI — Risk, control and adoption

- 28 Where your data goes

- 29 The governance layer

- 30 Data readiness and maturity

- 31 Your first 90 days

Reference

- 32 Frequently asked questions

- 33 Glossary

- 34 Sources

01 · Ground floor

## What a language model actually is

A language model predicts the next token; everything else you have heard about it is a consequence of doing that extremely well over a very large amount of text.

A language model looks far more mysterious from the outside than it does from the inside. At its core, it repeatedly estimates what token should come next. The surprising part is how much useful behaviour emerges once that prediction engine has absorbed enough examples of language, code, reasoning and human problem solving.

### Tokens first

Models do not read text as words. They split it into tokens, small units that may be a whole word, part of a word, punctuation or even a fragment of code. As a rough English rule, one token is about three quarters of a word, but the ratio varies by language and content. That is why a 10,000 word document does not create a 10,000 token bill.

During inference, the model looks at the tokens already in view, produces a probability distribution for the next one, selects a token, appends it, then does the same thing again. That simple loop is why generated text arrives incrementally.

### Where attention fits

The mechanism that makes the prediction useful is attention. Think of a meeting table covered with notes. For each new word it is about to produce, the model decides which notes on the table matter most. A reference to “it” may pull attention towards a product name three paragraphs earlier. A calculation may focus on the figures and operators nearby. A contract question may put most of its weight on the clause that defines the term being discussed.

This is the heart of the transformer architecture. Attention does not give the model unlimited memory and it does not guarantee that the right passage will dominate. It simply lets the model weigh relationships between the material that is currently available.

[Figure] Attention is selective weighting, not memory. For each step, the transformer can place different weight on different parts of the material in view. This sketch is conceptual, not a trace from a real model.

### Knowledge is not a database

A trained model contains an enormous number of learned parameters. Those parameters capture statistical structure from its training data, but they are not rows in a database that the model can query. A model may reproduce a fact because the pattern is strongly represented in its weights, yet it cannot point to a record inside itself and say, “this is where I stored it”.

That distinction explains hallucination, also called confabulation. The same generation mechanism produces both a correct answer and a plausible but false one. Fluency is not evidence of truth. A model can be extremely confident in its wording while the underlying claim is wrong.

Current information is also external. A model's training and post training happen over defined periods. If an assistant tells you about something that happened yesterday, the surrounding product has usually fetched fresh information through search, a database, a tool or another service and supplied that information to the model. The model did not acquire yesterday's news spontaneously.

### The context window: the model's working desk

The context window is everything available to the model for the current inference: your instruction, relevant conversation history, retrieved passages, tool descriptions, files, structured records and, in multimodal systems, representations of images, audio or video. It is measured in tokens.

The best analogy is a working desk, not a filing cabinet. Material on the desk can influence the answer. Material elsewhere cannot, unless the application fetches it. Modern systems may advertise context windows from hundreds of thousands to around two million tokens, but maximum capacity is not the same as useful capacity.

Long contexts introduce three costs at once: more input tokens, more latency and more opportunities for relevant information to be diluted by irrelevant material. Research on long context behaviour repeatedly shows variants of the same problem, often called context rot or lost in the middle effects. A larger desk is useful. Covering every centimetre of it with paperwork is not.

[Figure] Capacity is not a target. Long-context research shows that position, noise and task design can all affect performance. The practical lesson is to optimise signal density rather than assume that filling the advertised window improves the answer.

Memory is a system feature
The model can only reason over information made available to the current inference. An application may rebuild that context itself, or a provider may store conversation state and reconstruct it server side. Either way, durable memory comes from storage, retrieval and policy around the model, not from a permanent personal memory hidden in the weights.

### How a base model becomes an assistant

“The model” is actually the result of several stages. Knowing the stages removes a lot of confusion about fine tuning, alignment and why two models of similar size can behave very differently.

- Pretraining. The model learns broad statistical structure from very large datasets by predicting missing or next tokens. This creates general language and world knowledge, but not necessarily a useful assistant.

- Instruction and post training. The base model is trained on examples of questions, answers, tool calls and preferred behaviours so that it follows instructions rather than merely continuing text.

- Preference or reinforcement training. Human or model generated feedback is used to make useful behaviours more likely and undesirable behaviours less likely. Different laboratories use different techniques.

- Specialisation. Fine tuning, adapters, distillation or domain specific post training can change how the model behaves for a particular job.

- Inference. This is the stage you pay for each time the trained model is run. Prompting, retrieval, tools and agent loops all happen here, around an already trained model.

[Figure] Different layers solve different problems. Pretraining creates broad capability, post-training makes it usable, task adaptation changes recurring behaviour, and the surrounding system supplies current knowledge, permissions and actions.

This gives you a useful decision rule later in the guide: use prompting to tell the model what to do now, use retrieval to give it changing facts, use tools to let it act on systems, and use fine tuning when you need to change behaviour consistently across many examples. Fine tuning is usually a poor substitute for a knowledge base.

### Reasoning and “thinking”

Reasoning models spend additional inference on intermediate work before producing the final response. Depending on the provider, that work may be called reasoning, thinking, deliberation or a reasoning budget. Mechanically it is still model inference, but allocating more computation to intermediate steps can materially improve performance on maths, coding, planning and multi stage analysis.

The trade is real. More reasoning usually means more tokens or compute, more latency and more cost. It is not a universal quality switch. For extraction, faithful summarisation, routing and tightly grounded answers, additional reasoning may add little and can sometimes encourage the model to infer beyond the supplied evidence. Test reasoning settings against the task rather than assuming that more is always better.

Rule of thumb

### Temperature, top p and top k

Once the model has scored possible next tokens, the serving system still needs to choose one. Sampling settings shape that choice.

Temperature changes how sharp or flat the probability distribution is. Lower values concentrate probability on the leading candidates. Higher values give less likely candidates more chance. In practice, lower settings tend to produce more consistent wording, while higher settings produce more variation.

Top p, or nucleus sampling, keeps the smallest set of candidates whose cumulative probability reaches the chosen threshold. The set can be tiny when the model is confident and much larger when it is uncertain.

Top k simply keeps the k highest scoring candidates. It remains common in local inference servers even though many hosted APIs emphasise temperature and top p instead.

| Task | Useful starting point | Reasoning |

| Extraction, classification, routing | Low sampling, often temperature 0 to 0.2 | Usually minimal. Validate the output instead. |
| Grounded RAG answer or summary | Low sampling | Use only if your evaluation set shows a gain in faithfulness or completeness. |
| Drafting and ideation | Moderate to higher sampling | Useful when the task genuinely benefits from alternatives or planning. |
| Agent tool selection | Usually conservative sampling | Spend reasoning where planning is hard, not where a deterministic rule will do. |

Three practical cautions
Temperature zero does not guarantee identical runs. Backend changes, floating point behaviour, batching, model updates and mixture of experts routing can still create variation. If reproducibility matters, enforce it with schemas, deterministic checks and tests around the model.

Do not tune every sampler at once. Change one control, measure the effect, then decide whether another change is justified.

Randomness is not insight. Raising temperature can create variety; it does not make the model wiser. Better source material, examples, tools and evaluation usually matter more.

In plain English
A language model is a powerful prediction engine with learned knowledge but no dependable internal database, no permanent personal memory and no independent guarantee that what it says is true. Production AI becomes reliable by surrounding that engine with context, retrieval, tools, permissions, validation and a controlled loop. That surrounding engineering is the subject of the rest of this guide.

02 · Reality check

## When the answer is not a language model at all

Forecasting, anomaly detection and optimisation are usually solved better, cheaper and more reliably by methods that predate the chatbot.

Before asking which model to use, ask whether a language model belongs in the solution. In manufacturing and engineering, many valuable AI problems are still better solved by classical machine learning, computer vision or optimisation.

### AI is bigger than LLMs

Machine learning was improving factories long before chatbots reached the boardroom. Forecasting, anomaly detection, predictive maintenance, process control and visual inspection all existed as mature disciplines. Generative AI made one branch of the field conversational, then everyday language started using “AI” and “LLM” as if they meant the same thing.

That shortcut is expensive. If the signal lives in a vibration trace, a thousand production records or a constrained scheduling problem, asking a general language model to guess the answer is often a step backwards.

### A better first question

Ask: what kind of information contains the signal? Language, documents and loosely structured text point towards language models. Numerical series, sensor streams and tabular features usually point towards statistical or machine learning models. Pixels may call for computer vision. Hard constraints and objective functions point towards operations research.

This is not an absolute boundary. Modern multimodal models can inspect images, and language models can call code that analyses tables. The point is to put the right engine behind the interface rather than forcing the interface to become the engine.

| Business question | Technique to consider first | Why |

| What will demand be next quarter? | Time series or supervised forecasting | You need numerical extrapolation, back testing and uncertainty, not a plausible sentence containing a number. |
| Which machines are likely to fail? | Anomaly detection, survival models, condition monitoring | The useful signal is in measurements and event history. |
| Is this component defective? | Purpose trained computer vision, often classification or segmentation | Fast, repeatable image decisions are normally better handled by a model trained for the inspection task. |
| Which customers are at risk of leaving? | Gradient boosted trees or another supervised tabular model | Structured features and a known target are exactly where classical ML remains strong. |
| What is the best production schedule? | Constraint programming, mixed integer optimisation or heuristics | The problem is defined by constraints and an objective function. Use a solver to optimise it. |
| Is this payment suspicious? | Rules plus statistical or anomaly models | Latency, traceability and stable decision thresholds matter. |
| What themes are customers complaining about? | Language model | The signal is in natural language, often across many loosely written cases. |
| What does this 200 page standard require from us? | Language model with retrieval | The work is reading, locating evidence, synthesising and explaining. |

[Figure] A first routing question, not a rigid taxonomy. Modern systems can cross these boundaries, but production designs are usually stronger when specialist numerical methods do the detection or optimisation and the language model handles explanation, retrieval and workflow.

### The pattern that often wins in industry

The strongest design is frequently a hybrid. Let the specialist model calculate or detect. Let the language model explain, retrieve and orchestrate.

A forecasting model can produce the demand estimate and prediction interval; the language model can explain the drivers and draft the planning note. An anomaly detector can flag a bearing; the language model can pull service history, find the relevant maintenance instruction and prepare a work order. A vision system can detect the defect; the language model can draft the non conformance report and retrieve similar historical cases.

That division of labour avoids two common failures: using a language model where a numerical method is more reliable, and building an excellent numerical model whose output never reaches the person who needs to act.

A useful test in a scoping meeting
Imagine a very capable analyst joining the company tomorrow. Would they solve the problem mainly by reading and interpreting material, by calculating from measurements, by looking at images, or by satisfying a set of mathematical constraints? The answer usually points towards the right technical family. If the job requires several of those activities, design a hybrid rather than searching for one model to do everything.

### Small models still create large returns

A well trained gradient boosted model on your own historical data may run on a laptop, cost almost nothing per prediction and outperform a frontier general model on a narrow tabular problem. It can also be easier to validate, monitor and explain. It will never create a spectacular chatbot demo. That does not make it less valuable.

03 · The map

## Four layers around the model

The prompt, the context, the execution layer and the loop: a chatbot uses two of them, a production agent needs all four.

AI terminology changes quickly because the bottleneck keeps moving. Prompt engineering, context engineering, harness engineering and agent loops are not rival schools. They describe four different layers of the same production system.

During the first chatbot wave, instruction following was fragile enough that wording dominated results. Teams spent their effort on prompts. As models improved, the larger problem became information: which documents, memories and examples should enter the context? That is context engineering.

Once models became dependable tool users, attention shifted again. Which tools can the model call? What credentials do they hold? What can be written, deleted or sent? What validates the result? That surrounding execution environment is the harness.

Agents add the fourth layer: repetition. The system observes, acts, checks, decides what to do next and repeats until a stop condition is met. The quality of that loop, including its state, verification and failure handling, now matters as much as the intelligence of any individual model call.

| Layer | What you control | Typical failure | First place to look |

| Prompt | Instruction, examples, constraints, desired output | The model has the right information but produces the wrong shape, tone or decision | Clarify the instruction or add a worked example |
| Context | Documents, history, memory, records, tool descriptions | The answer is built on missing, stale or noisy information | Improve retrieval, filtering or memory selection |
| Harness | Tools, permissions, sandbox, validators, limits | The model chooses a sensible action but the environment is unsafe or unreliable | Reduce privileges, simplify tools, validate inputs and outputs |
| Loop | Planning, retries, state, verification, stop conditions, handoffs | The agent stops too early, loops indefinitely or loses progress | Add explicit state and an independent success check |

The commercial reason to use the model
When a pilot disappoints, teams often change the underlying model first. That is frequently the wrong diagnosis. Ask which layer failed. In our production work, missing context and weak verification are more common causes of failure than insufficient frontier intelligence. A model upgrade is useful only when the model is actually the bottleneck.

04 · Layer one

## Prompt engineering that still matters

Most of what survived from 2023 is simply writing a clear brief; the rest was compensating for models that have since got better.

Prompting is no longer the most exotic part of AI engineering. It is still the fastest way to improve an everyday task, and most users leave a large amount of quality on the table by treating the model like a search box.

### 1. Supply the briefing a colleague would need

If a competent stranger could not complete the task from what you wrote, neither can the model. “Draft a follow up to this customer” leaves the relationship, objective, prior agreement, urgency and house style undefined. The model will fill gaps because producing an answer is its job. Give it the relevant facts first.

### 2. Define the deliverable

Specify the shape of success. Length, format, audience, tone, mandatory fields and exclusions are all useful. “Summarise this” delegates too many decisions. “Give me five bullets, each under twenty words, separating facts from recommendations” is much easier to evaluate.

### 3. Use examples when form matters

Few shot prompting means showing input and output pairs that demonstrate what good looks like. Two or three representative examples can outperform a page of stylistic instruction. Choose typical examples first. A prompt filled with edge cases can teach the wrong pattern.

Field note

### A constraint register, carried into every prompt

On an eight-week market-entry engagement we ran the whole research pipeline from a numbered constraint file — sixteen rules plus an NDA register — prepended to each of forty-two engineered prompts rather than restated from memory. Ninety conversations later the constraints had not drifted, because there was only one place they lived.

### 4. Separate instructions from material

Use headings, fenced blocks, XML style tags or other clear boundaries when you provide source text. This improves readability for the model and gives you a basic defence against accidentally treating document content as instructions. It is not a security boundary, but it is good hygiene.

Field note

### What a good abstention looks like in production

On a weekly market intelligence system, every finding is re-read against the market it occurred in. Where the system cannot frame a finding in local context, the designed failure is to return it with no reframe — never to invent a plausible one. Across two hundred findings in five territories at acceptance, no stage degraded to a fabricated interpretation.

### 5. Give the model a legitimate way to abstain

This is one of the cheapest controls in applied AI. Tell the model what to do when the evidence is missing: “write not stated”, “ask a question”, “return insufficient evidence”, or “do not infer a value that is absent”. If completeness is the only visible objective, the model has an incentive to make the output look complete.

### 6. Use reasoning where the task earns it

Hard analysis, planning, coding and multi step comparison may benefit from explicit reasoning mode. Straight extraction, classification and formatting often do not. Use your evaluation set to decide. Reasoning is a resource allocation choice, not a badge of quality.

### 7. Iterate rather than worship the first prompt

Conversation is part of the interface. Correct the first answer precisely: “the recommendation is good, but the evidence is buried; put the decision and deadline first.” A few targeted turns are often more efficient than trying to write a mythical perfect opening prompt.

### 8. Tag uncertain inputs

If something you provide is an assumption, say so. For example: “We think this competitor uses halogen illumination. Source: an unverified trade show conversation. Check it and tell me how strong the evidence is.” Without that label, your assumption can silently become a premise for everything downstream.

A reusable pattern
# Role
You are reviewing supplier contracts for a UK manufacturing SME.

# Objective
Find every clause that creates a payment obligation.

# Output
Return a table with: clause reference | obligation | trigger | amount or formula.
If the document does not state a field, write "not stated".
Do not infer an amount that is absent from the source.

# Example
Input: "4.2 The Buyer shall pay a restocking fee of 15% on returns after 30 days."
Output: 4.2 | Restocking fee | Return after 30 days | 15% of order value

# Source material
<contract>
...paste here...
</contract>

The line that matters most
“If the document does not state a field, write not stated. Do not infer a value.” That instruction changes the objective from producing a complete looking answer to producing an evidence based one. In business workflows, that difference matters more than adding another paragraph of persona.

### What to stop spending time on

- Threats, bribes and emotional theatre. They are not a dependable 2026 optimisation technique.

- Huge fictional personas. A concrete functional perspective is useful; ornamental credentials are mostly tokens.

- Negative instructions alone. State what the model should do, not only what it must avoid.

- Assuming a prompt is permanent. Model updates change behaviour. Treat prompts as versioned system components and run regression tests when the model changes.

### Prompt, retrieval, fine tuning or tools?

| You need to change… | Reach for… | Example |

| What the model should do on this call | Prompting | Return a concise engineering risk review in a fixed format |
| What facts it can use | Retrieval or tools | Answer from the latest manuals, ERP records or customer history |
| What the system can execute | Tool use | Create a ticket, run a calculation, query stock, send for approval |
| A repeatable behaviour across many examples | Fine tuning or adapters | Specialised classification style, domain notation, constrained response behaviour |
| The cost or size of a capable model | Distillation | Train a smaller model to reproduce a stronger model on a defined task family |

[Figure] Do not use fine-tuning as a filing cabinet. Prompting changes the current instruction, retrieval supplies changing evidence, tools provide actions, and fine-tuning is for recurring model behaviour.

05 · Layer two

## Context engineering: deciding what the model gets to see

The scarce resource is not what the model knows but what it can see right now, and deciding that is an engineering job rather than a writing one.

The prompt is only one part of the request. Production quality depends on the whole package around it: history, documents, memory, tool definitions, records and intermediate results. Choosing that package is context engineering.

At model level, there is no magical continuity between calls. The model reasons over the context available to the current inference. Some applications resend or reconstruct conversation history themselves; some providers store conversation state and rebuild it server side. The implementation varies, but the engineering question is the same: what information should be in view now?

Context has three obvious costs. Tokens cost money. Larger inputs take longer to process. Excess material can reduce quality by competing for attention. Those pressures all favour the same discipline: give the model enough, not everything.

### Four operations cover most context work

| Operation | Purpose | Typical implementation |

| Write | Move information out of the context so it persists | Database state, task notes, a progress file, saved user preferences |
| Select | Bring in only what the next step needs | RAG, permission filters, record lookup, dynamic tool selection |
| Compress | Reduce the token footprint of material already collected | Summarising old turns, extracting relevant rows, replacing raw logs with a concise state |
| Isolate | Keep unrelated work from competing inside one window | Separate workers or sub agents that return compact findings to a coordinator |

[Figure] Context is a managed resource. These four operations recur across agent frameworks because they solve the same underlying problem: what deserves to be in the model's field of view now?

### Prefer just in time context

Large windows tempt teams to preload entire repositories. A better pattern is progressive disclosure: give the model references and ways to fetch detail when it becomes relevant. A file catalogue, record identifier, search function or database query tool is often better than dumping every possible source into the initial prompt.

This mirrors how experienced people work. They do not memorise the archive before starting. They know what exists, then retrieve the relevant folder when the question demands it.

### Memory has several meanings

- Conversation state. The turns that remain available to the current interaction. Useful, but increasingly noisy as the exchange grows.

- Long term memory. Facts or preferences stored between sessions and reintroduced later. The difficult part is not storage; it is deciding what deserves to be saved, updated or forgotten.

- Working state. The durable notes that let a long running task survive retries, context limits, process restarts or handoffs to another worker.

The fastest debugging question
Take a bad answer and ask: could a capable person have answered correctly from exactly the information the system supplied? If not, improve the context before rewriting the prompt or buying a stronger model. If yes, then investigate instruction quality, model capability or the verification layer.

06 · Layer two, applied

## RAG: letting the model answer from your information

Retrieval turns a memory test into an open-book exam, which is why the quality of the book matters more than the cleverness of the student.

Retrieval augmented generation, or RAG, is the standard pattern for answering from documents and private knowledge without retraining a model every time the source material changes. The idea is simple. Retrieval quality is not.

### Open book instead of memory test

Suppose you want an assistant to answer from service manuals, contracts, policies and historical cases. Teaching those changing facts through model training is usually the wrong approach. Instead, find the relevant evidence at query time, place it in context and ask the model to answer from that evidence.

That is the core RAG loop: retrieve, then generate.

### A production pipeline, step by step

- Ingest and parse. Turn PDFs, web pages, emails, records or other sources into usable text and metadata. Preserve headings, tables, page references and access controls where possible.

- Chunk. Split content into retrievable units. Structure matters more than arbitrary character counts. A clause, troubleshooting procedure or coherent section is usually more useful than a blind slice through the middle of a sentence.

- Represent. Create embeddings for semantic search and, where useful, a keyword index for exact terms.

- Retrieve. Search for a broad candidate set with filters for tenant, user permissions, document status, product, date or other metadata.

- Rerank. Re score the candidate set with a model that reads query and passage together. This often improves which evidence reaches the top of the list.

- Generate. Pass a small, high signal evidence set to the language model and require citations or source identifiers for material claims.

- Evaluate. Measure retrieval and answer faithfulness separately. A polished final answer cannot tell you which stage failed.

[Figure] A common production shape, not the only one. Indexing prepares the corpus ahead of time. At query time, retrieval optimises for finding the evidence, reranking improves order, and generation receives a much smaller grounded set.

### Three upgrades that often pay quickly

Hybrid retrieval. Dense embeddings are good at meaning but can miss exact strings such as SKUs, error codes, clause numbers and names. Keyword retrieval has the opposite strengths. Running both and fusing the rankings is a strong general starting point for mixed business corpora.

Reranking. If the correct passage is usually somewhere in the candidate set but not near the top, a reranker is often a high return improvement. It is not a cure for poor first stage recall.

Contextual chunks. Add enough document context to make an isolated passage meaningful before embedding it. A chunk saying “the maximum is £5,000” is weak. A chunk saying it comes from the 2026 travel policy, section 4, makes the same sentence much easier to retrieve correctly.

Rule of thumb

### Permissions and deletion are part of RAG

A production index must respect the permissions of the source system. If an employee cannot open a document in SharePoint, your assistant should not surface its contents because the vector database happens to contain an embedding. Preserve tenant and access metadata at ingestion, apply it before or during retrieval, and test permission boundaries explicitly.

Deletion matters too. Removing a source file should trigger removal of its chunks, vectors, cached answers and derived metadata. Otherwise you create a knowledge system that remembers material the organisation believes it has deleted.

[Figure] Keep the lineage from source to every derivative. A stable identifier lets you block retrieval at once, then clean the downstream representations and prove that no live path still returns the deleted information.

### When RAG is the wrong tool

- Everything already fits comfortably in context. For a small, stable corpus, direct context can be simpler and more reliable than adding a search layer.

- The answer is a database calculation. “Revenue by region in Q3” belongs in SQL, a semantic layer or another structured query tool.

- The answer depends on explicit relationships. Ownership chains, dependencies and sanctions exposure may need graph traversal or another structured representation rather than similarity search alone.

- The source changes every second. Query the system of record directly if freshness matters more than semantic search over a periodically refreshed index.

### Long context and retrieval are complementary

Large context windows reduce pressure on retrieval, but they do not eliminate the need to select information. Feeding a million tokens into every question costs more, takes longer and gives the model more irrelevant material to navigate. Long context is valuable because it lets you provide richer evidence when necessary, not because it makes relevance disappear as an engineering problem.

### Agentic RAG

Fixed RAG performs one retrieval step before answering. Agentic RAG gives the model a search capability and lets it decide whether to search again, change the query, inspect another source or call a different tool. That flexibility helps on ambiguous and multi source questions, but it also adds latency, tokens and failure modes. A mature system routes simple questions through a short path and reserves the loop for cases that need it.

Evaluate the search before the answer
Build a labelled set of real questions and identify the evidence that should be retrieved for each one. Measure whether the search stage finds it. If the necessary passage never reaches the candidate set, prompting cannot recover it. This separation between retrieval evaluation and generation evaluation is one of the most important habits in production RAG.

07 · Taxonomy

## Nine practical RAG patterns, and why other lists give different counts

There is no canonical number of RAG architectures; choose the pattern from the failure you are actually seeing, not from a list.

Search for “types of RAG” and you will find lists of three, nine, twelve or more. Most of the disagreement comes from counting different things. Academic surveys describe broad evolutionary paradigms; research papers name individual methods; practitioners usually care about architectural patterns they can choose for a real system.

### Three levels of vocabulary

An influential survey popularised three broad stages: Naive RAG, Advanced RAG and Modular RAG. Treat those as a useful historical framing rather than a formal industry standard. Research literature then contains many named variants, often tied to individual papers.

This guide uses a different level: nine practical patterns. It is our working taxonomy for architecture discussions. The aim is not to win a naming argument; it is to connect a recognisable failure mode to a design response.

| Broad paradigm | Typical shape | What pushed the field further |

| Naive | Retrieve once, then generate | Weak recall, noisy evidence and no adaptation to difficult questions |
| Advanced | Better ingestion, query transformation, hybrid retrieval, reranking and filtering | The pipeline is still largely fixed regardless of question complexity |
| Modular | Components can be routed, repeated, skipped or combined | More flexibility brings more state, cost and operational complexity |

### What businesses usually mean by RAG

Strip away the acronyms and most enterprise RAG projects are trying to build the same thing: a conversational interface that can answer from the organisation's own material and show where the answer came from. The sophistication below exists because doing that reliably is harder than making the demo look convincing.

### The nine patterns in this guide

1. Baseline RAG
One retrieval step, then one answer. Build it first because it gives you a measured baseline. Every extra component should justify itself against that baseline.

2. Hybrid RAG
Semantic retrieval and lexical retrieval run together, then their rankings are fused. This is particularly useful in technical and business corpora where users mix natural language with exact identifiers, names, part numbers and references.

3. Multi hop or iterative RAG
The system retrieves, learns something, formulates a better next query and retrieves again. Use it when no single passage contains the full answer and the solution must be assembled across sources.

4. Corrective RAG
A separate check grades the retrieved evidence before generation. If the evidence is weak, the system can search again, switch source, ask for clarification or abstain. This is valuable where unsupported confidence is more expensive than an explicit “not enough evidence”.

5. Self RAG
Self RAG refers to the specific research approach in which retrieval and reflection behaviour are trained into the model. Do not use the name for every workflow that critiques its own search. For the generic idea, “self reflective retrieval” is clearer.

6. Adaptive RAG
A router chooses a path based on the question. A simple lookup may use one cheap retrieval pass; a difficult cross document question may receive iterative or agentic treatment. This can reduce cost on mixed traffic, although many successful systems never need more than a well measured hybrid pipeline.

7. Agentic RAG
Search becomes a tool inside an agent loop. The model decides what to search, inspects the result, judges sufficiency and can call other tools before answering. Use the flexibility when the path genuinely cannot be known in advance.

8. Graph based RAG
This is a family of methods that combines retrieval with explicit entities and relationships. It earns its keep on questions about dependency, ownership, hierarchy or connections that similarity search alone cannot answer cleanly.

9. Multimodal RAG
Relevant evidence may live in diagrams, tables, screenshots, scans, drawings, audio or video as well as prose. Multimodal RAG can use shared multimodal embeddings, modality specific indexes, OCR and table extraction, or combinations of them. The architecture depends on the material.

What about full context generation?
If the whole knowledge base is small enough, you can place it directly in context and skip retrieval. That is a legitimate technique, sometimes called cache augmented or full context generation, but it is not a form of RAG. A router can still choose between direct context and retrieval depending on corpus size, cost and question type.

### Choose from the failure, not the label

| Observed problem | Design response to test |

| You have no measured baseline | Baseline RAG |
| Names, codes and references are missed | Hybrid retrieval and exact fields |
| The right evidence appears but ranks badly | Reranking, which is a technique rather than a separate RAG family |
| The answer requires several documents | Multi hop retrieval |
| The system answers when retrieval is weak | Corrective checks and abstention |
| Easy and difficult questions cost the same | Adaptive routing |
| The research path is genuinely open ended | Agentic RAG |
| The question is relational | Graph based retrieval |
| Important evidence is visual or tabular | Multimodal ingestion and retrieval |

The architecture is successful when it fixes a measured failure. Nobody on the board needs to care whether the team calls it corrective RAG. They need to know that the system now declines to answer when it cannot find reliable evidence.

08 · Vector search

## Embeddings and vector search, without the magic

An embedding puts meaning on a map, so that “near” becomes something a computer can calculate.

Embeddings are the mechanism behind semantic retrieval. They turn meaning into coordinates that a computer can search quickly. Once you understand the trade offs, vector search stops looking like a black box and starts looking like ordinary information retrieval engineering.

### What an embedding is

Think of a library that shelves by what a book is about rather than by title or author. Two books nobody would file together end up side by side because they cover the same ground — and “nearby” becomes something you can measure.

An embedding model converts an input into a fixed length vector. A 1,024 dimensional embedding contains 1,024 numbers. Training encourages semantically related inputs to occupy useful relative positions in that space, so a question about “termination notice” can find a clause written as “either party may terminate on ninety days' written notice” even when the wording differs.

The library analogy works well: instead of shelving documents alphabetically, you shelve them by meaning. The vector is the shelf coordinate.

[Figure] Similarity is a property of the embedding model and metric. Cosine similarity is a common example, but scores are not calibrated across embedding models. Always choose and threshold on your own labelled data.

### Similarity is model dependent

Cosine similarity compares the angle between two vectors. Mathematically it ranges from minus one to one, although real embedding models often occupy a much narrower range. Zero means orthogonal, not universally “unrelated”.

Dot product becomes equivalent to cosine ranking when vectors are appropriately normalised. Euclidean distance measures straight line separation. The practical rule is simple: use the similarity or distance function recommended for the embedding model and index rather than assuming one metric is always best.

### Dimensions trade quality against storage and speed

A float32 vector uses four bytes per dimension. A 768 dimensional vector therefore occupies about 3 KB before metadata and index overhead. Ten million such vectors are about 30 GB of raw vector data.

More dimensions can preserve more information, but they also increase memory, storage and search cost. Some modern embedding models use Matryoshka representation learning, where earlier dimensions carry more of the useful signal and the vector can be truncated with graceful quality loss. Do not assume the feature exists, and do not assume the same truncation ratio is safe across models. Test it on your retrieval set.

### Choosing the model

MTEB and similar leaderboards are useful for narrowing the field, not for making the final decision. Check language coverage, input length, licence, embedding dimensions, latency, hosting options and domain behaviour. Then run your own labelled questions. A model that ranks slightly lower overall can outperform the leader on your technical vocabulary.

Changing embedding model later means re embedding the corpus because vector spaces from different models are not interchangeable. That migration can be trivial for 20,000 chunks and material for tens of millions. Treat the choice as an architectural dependency, not as a permanent marriage.

### Exact search versus approximate search

Exact nearest neighbour search compares the query with every candidate vector. Approximate nearest neighbour, or ANN, avoids most comparisons and accepts a small, measurable recall trade for much higher throughput. The point at which exact search becomes impractical depends on dimensions, hardware, filters, query volume and latency targets; there is no universal vector count where it suddenly stops working.

### HNSW: the graph index you will meet everywhere

Hierarchical Navigable Small World builds a layered graph. Upper layers allow large jumps; lower layers refine the neighbourhood. A useful analogy is travelling by motorway, then local road, then street, instead of checking every address in the country.

[Figure] Why HNSW is fast. The index uses sparse upper layers to move quickly towards a promising region, then denser lower layers to refine the search. Exact complexity depends on data and parameters; this diagram shows the navigation idea, not a performance guarantee.

Implementations expose related controls with slightly different names and defaults. Three concepts matter:

| Control | When it matters | What raising it usually does | pgvector example, Aug 2026 |

| m | Index build | More graph connections, usually better recall, more memory and build work | Default 16 |
| ef_construction | Index build | More candidate exploration while constructing the graph | Default 64 |
| ef_search or equivalent | Query time | Broader search, normally improving recall at a latency cost | Default 40 |

Query time breadth is especially useful because you can change it without rebuilding the index. A low latency interactive path and a nightly high recall analysis can use different settings. Always check your database documentation because defaults are not portable between pgvector, Qdrant, Weaviate, Milvus, Elasticsearch and other engines.

### IVF and k means

An inverted file index, or IVF, partitions vector space into clusters, often using k means. At query time the system first finds nearby cluster centroids, then searches only selected lists. The nprobe control determines how many lists are inspected.

The trade off is intuitive. Probe too few clusters and a relevant neighbour across a boundary can be missed. Increase nprobe and you recover recall by doing more work. IVF remains useful at large scale and where build cost or compression matters.

K means also appears in product quantisation codebooks and offline corpus exploration. For near duplicate detection, direct similarity thresholds are often simpler than clustering.

### Four query time controls to recognise

| Control | Common names | Purpose |

| Result count | k, top_k, limit | How many candidates the index returns |
| Relevance floor | score_threshold, minimum similarity, maximum distance | Return nothing below a derived quality threshold |
| Search breadth | ef_search, hnsw_ef, nprobe | How hard the ANN index searches before returning results |
| Metadata filter | payload filter, WHERE, tenant or ACL filter | Restrict candidates by permission, customer, date, type or other fields |

Rule of thumb

### Retrieve broadly, show narrowly

The number of candidates retrieved is not the number of passages you should place in the model context. An Ascentis starting pattern for many document systems is to retrieve tens of candidates, rerank them, then send a much smaller evidence set to generation. Exact values depend on corpus size, chunking, latency and the reranker.

### Thresholds must come from your data

A cosine score of 0.75 has no universal meaning. Score distributions change with embedding model, domain and chunk length. Build positive and negative examples from your labelled set, plot their distributions and choose a threshold that matches your desired balance between abstention and recall.

### Filters can damage recall

Multi tenant and permission aware retrieval adds an important complication. A graph built over the whole corpus may behave differently when only a tiny filtered subset is eligible. Modern engines offer filter aware strategies and payload indexes, but you still need to measure retrieval with the real filters applied. Unfiltered benchmark quality tells you little if every production query carries an ACL.

09 · Fusion

## Hybrid retrieval and score fusion

Dense and keyword search fail on different queries, and fusing their rankings is cheaper than making either one perfect.

Semantic search finds similar meaning. Lexical search finds matching language. Real business queries contain both, which is why hybrid retrieval is such a strong default starting point for technical, legal and support knowledge bases.

### Where dense retrieval struggles

An embedding compresses a passage into one vector. That is excellent for paraphrase and weak for some exact clues. “What is our remote working policy?” is semantic. “INV 4471”, “E2214”, “7.4(b)” and a specific part number are lexical.

BM25 and related keyword methods remain extremely useful. For technical identifiers, go one step further: keep an exact or raw field so tokenisation does not split the very string you need to match.

Rule of thumb

### Reciprocal Rank Fusion

Vector and keyword scores live on different scales, so adding raw scores is usually meaningless. Reciprocal Rank Fusion, or RRF, avoids that problem by using rank positions rather than raw scores.

RRF(d) = Σ 1 / (k + rankᵢ(d))
A document that appears reasonably high in both lists can beat one that a single retriever loves but the other does not find. In effect, RRF rewards consensus.

The constant k controls how strongly top ranks dominate. A value of 60 is a common convention from information retrieval practice, not a law. On short candidate lists, smaller values can create more separation. Tune the value against labelled queries rather than inheriting 60 without testing.

Major search platforms support RRF or equivalent rank fusion, which makes it a practical place to start when you do not yet have enough labels to calibrate score weights.

[Figure] Why RRF is a sensible baseline. Dense and lexical retrievers score on different scales. RRF avoids pretending those scores are directly comparable and rewards documents that rank well across lists.

### Weighted fusion

The alternative is to normalise both score sets and calculate a weighted combination.

score(d) = α × semantic_score(d) + (1 − α) × lexical_score(d)

| Illustrative α | Bias | Traffic shape where you might test it |

| 0.75 | Semantic | Natural language questions over prose |
| 0.50 | Balanced | Mixed conceptual and exact lookup traffic |
| 0.25 | Lexical | Part numbers, case references, names and technical identifiers |

Those values are starting points only. Weighted fusion can beat RRF on a calibrated dataset, and it can also lose badly when score normalisation or query mix changes. If you introduce weights, keep the evaluation set and rerun it as the corpus evolves.

Look at your query distribution before tuning anything
Take a few hundred real searches and classify them as conceptual, exact identifier or mixed. Support and legal systems often contain more exact lookup than teams expect. That one exercise can explain why a dense only system looks brilliant in a demo and frustrating in production.

### Query transformation

- Query rewriting. Resolve pronouns, conversational shorthand and missing entities before retrieval. “What about the second one?” may need to become “What is the notice period in the Thompson supply agreement?”

- Query expansion. Generate several plausible phrasings, retrieve for each and fuse the results. Useful when recall is limited by user wording.

- HyDE. Generate a hypothetical answer or document, embed that representation and search with it. It can help on some prose retrieval tasks, but performance is task dependent and exact identifier queries are a poor fit.

10 · Precision

## Rerankers: a second opinion on the shortlist

Retrieval finds the evidence; reranking decides what the model actually reads.

A reranker is one of the first components we test when retrieval finds the right evidence but places it too low. It solves a different problem from the first stage retriever, which is why the combination is powerful.

### Bi encoder versus cross encoder

It is the difference between sorting a stack of CVs by keyword and actually reading two of them next to the job description. The first is fast enough to run on everything; the second is accurate enough to decide.

A conventional dense retriever is a bi encoder. It encodes the query and document separately, which allows documents to be indexed in advance and millions of candidates to be searched quickly.

A cross encoder reads the query and candidate passage together. That joint view often produces much better precision on a small candidate set because the model can directly evaluate whether this particular passage answers this particular question. It is too expensive to run over the whole corpus, so it operates after fast retrieval.

| | First stage retrieval | Reranking |

| Query and passage | Usually represented separately | Read together |
| Scale | Potentially millions of candidates | A shortlist |
| Primary objective | High recall, do not lose the answer | High precision, put the best evidence first |
| Cost | Very low per candidate | Higher per candidate |

The distinction is worth remembering: retrieval tries not to miss; reranking tries not to waste the context window. A reranker cannot recover a passage the first stage never retrieved.

[Figure] Sift, then read closely. A common starting point is dozens of candidates into a reranker and a single-digit or low-double-digit evidence set out. Tune those numbers on your corpus rather than copying them blindly.

Rule of thumb

### Configuration as a starting point

For many document systems we begin with dozens of candidates into the reranker and a single digit or low double digit evidence set out. The familiar “50 to 100 in, 5 to 15 out” range is a useful engineering starting point, not a rule. Chunk size, corpus size, latency target, reranker limits and query difficulty all change the optimum.

Managed reranking APIs and open weight cross encoders are both viable. Late interaction models such as ColBERT occupy another point in the design space, retaining token level representations for more expressive matching at a higher storage cost than standard dense retrieval.

A strong baseline pipeline
Hybrid retrieval → rank fusion → reranking → small evidence set → derived relevance floor → grounded generation with citations. We use this as a baseline because it is easy to measure and surprisingly hard to beat with elaborate architecture unless the corpus genuinely needs graph, multimodal or agentic behaviour.

11 · Measurement

## Retrieval metrics that tell you where the system failed

If you cannot say whether the failure was in finding the evidence or in using it, you cannot fix either.

A wrong final answer cannot tell you whether the search failed, the reranker buried the evidence or the generator ignored good context. Retrieval needs its own measurements.

| Metric | Question | Why it matters |

| Recall@k | What fraction of all relevant items appeared in the top k? | Measures whether the first stage found the required evidence |
| Hit@k | Did at least one gold relevant item appear in the top k? | Useful when each question only needs one target passage |
| Hit@1 | Is the top ranked result relevant? | Simple way to inspect top rank quality after reranking |
| Precision@k | What fraction of returned items are relevant? | Shows how much noise you are sending downstream |
| MRR | How high does the first relevant result usually appear? | Reciprocal rank gives 1.0 for first, 0.5 for second, 0.2 for fifth |
| nDCG@k | How good is the ranking when relevance is graded? | Accounts for both relevance strength and position |

### A simple diagnostic pair

Measure a broad recall or hit metric before reranking, then a top rank metric after reranking. The combination quickly localises the problem.

| First stage recall | Top rank quality | Likely diagnosis |

| Low | Low | The evidence was never found. Investigate ingestion, chunking, embeddings, filters and hybrid retrieval. |
| High | Low | The evidence exists in the candidate set but ranking is weak. Investigate reranking. |
| High | High | Retrieval is probably healthy. Look at generation, instructions, context use or verification. |

[Figure] Measure retrieval separately from generation. The exact k depends on the task. The useful habit is to pair a broad first-stage metric with a post-rerank top-rank metric so you know which component to fix.

MeasuredField note

### Published numbers from Cortex

To show what measurement looks like in practice, Ascentis AI published a locked benchmark for Cortex, our self hosted support intelligence engine. In that evaluation:

- Hit@5: 83.3%. Five of six benchmark questions had the gold answer passage within the top five retrieved results. We previously labelled this Recall@5; Hit@5 is the more precise name for that methodology.

- Citation rate: 95.8%. This is a generation or system metric, not a retrieval metric. It measures the share of answer claims carrying a source reference.

- Measured fabrication rate: 5.0%. Again a system level measure, published because a known residual error rate is more useful than an unmeasured one.

Do not compare those numbers directly with another product unless the evaluation protocol matches. The reusable lesson is methodological: ask for question count, corpus version, gold label process, reranking state, model version and evaluation date.

### Targets are local

Thresholds such as 90% recall or 75% Hit@1 can be useful internal starting points on a well behaved corpus, but they are not industry standards. A legal research system, a support manual assistant and a product catalogue have different ambiguity and different costs of missing evidence.

Rule of thumb

### Build the labelled set early

Fifty to one hundred real questions are enough to start regression testing, not enough to claim complete coverage of a complex product. Pull them from support tickets, search logs and domain experts. A model can propose candidate labels; a human should confirm the gold evidence. Keep the set versioned. It will outlive your current embedding model, database and generation model.

The procurement question
Ask an AI supplier for retrieval and generation metrics together with the methodology used to produce them. A polished demo shows that a system can work. A reproducible evaluation shows that the supplier knows when it stops working.

Related on this siteA worked example with published retrieval numbers on a real deployment: the market-intelligence case study.

12 · Structure

## Semantic retrieval versus explicit relationships

Similarity answers “what sounds related”; a graph answers “what is connected, and how” — and no amount of embedding quality turns the first into the second.

Embeddings are very good at finding things that mean similar things. They are not an auditable database of who owns what, which rule supersedes another, or which component depends on which assembly.

Vector representations can encode relational patterns implicitly, but similarity does not give you an explicit typed relationship that you can traverse, constrain and inspect. “Buyer indemnifies seller” and “seller indemnifies buyer” may be semantically close while assigning opposite obligations. Direction matters.

### Three terms worth separating

- Taxonomy. A hierarchy of categories, such as product family → product line → model.

- Ontology. A formal definition of entity types, relationships, constraints and meanings.

- Knowledge graph. Concrete entities and relationship instances stored according to a schema or ontology.

| | Semantic / vector | Ontology / graph |

| Represents | Similarity and implicit semantic structure | Explicit entities, types and relationships |
| Strong at | Paraphrase, fuzzy questions, synonymy | Hierarchy, direction, dependency, constrained multi hop traversal |
| Created by | Embedding model and ingestion pipeline | Existing business systems, domain modelling and sometimes AI assisted extraction |
| Failure mode | Similar wording can hide opposite meaning or missing structure | Anything omitted or wrongly modelled becomes structurally wrong |
| Maintenance | Re embed changed content | Govern the schema and update entity relationships |

[Figure] Embeddings do not replace typed relationships. Vector search is excellent for semantic discovery. Graphs and ontologies earn their cost when the question depends on explicit ownership, dependency, hierarchy or provenance.

Field note

### Same name, different company

A competitor monitor we built kept surfacing people and organisations that merely shared a surname with the entities being tracked. Similarity cannot tell a namesake from a match, so the fix was an entity verification gate using explicit identifiers, not a better embedding model. It screened fourteen look-alikes out of a single client's digest.

### When graph based retrieval earns the effort

Graph based RAG can extract entities and relationships from documents, connect them with structured reference data and retrieve by graph traversal as well as semantic similarity. It is useful for dependency networks, contractual relationships, product structures and other questions where the path between entities matters.

It also creates a new source of error. If a language model extracts a relationship incorrectly and the graph treats that edge as fact, the mistake becomes structured and easy to reuse. Where relationships drive business decisions, AI assisted extraction still needs domain governance and provenance.

### Start with structures you already own

Most companies already maintain useful taxonomies and graphs without using those names: the bill of materials, product hierarchy, chart of accounts, CRM relationships, organisation chart and client matter structure. Connecting retrieval to those existing structures is usually cheaper and safer than inventing a knowledge graph from scratch.

Related on this siteThis distinction is the basis of Cortex, the Ascentis AI retrieval product.

13 · Storage

## Vector databases: use the simplest one that meets the workload

Pick the simplest store that meets the workload, because vector count is almost never the constraint that actually bites.

A vector database stores embeddings, builds an ANN index, applies metadata filters and returns nearest neighbours under concurrent load. The “best” choice depends more on your existing estate and workload than on a generic leaderboard.

| Option | Shape | When it is worth considering |

| pgvector | Postgres extension | You already run Postgres and want vectors, metadata, joins and transactions in one operational system |
| Qdrant | Open source dedicated vector engine, self host or cloud | Strong filtering, sparse and multi vector features, dedicated retrieval workload |
| Weaviate | Open source or managed | Hybrid retrieval and integrated search features are central |
| Milvus / Zilliz | Distributed vector platform | Very large scale and a team comfortable operating distributed infrastructure |
| Pinecone | Managed service | You value developer speed and minimal operations more than infrastructure control |
| Elasticsearch / OpenSearch / Vespa | Search platforms with vector capability | You already operate search and want lexical, vector and filtering in one platform |

The deliberately boring recommendation
If Postgres already meets your retrieval performance, filtering and scale requirements, start with pgvector and move on. A specialist database should solve a measured problem, not create a second backup regime and consistency problem because the category sounds more AI native.

### Vector count alone is not the sizing rule

You will sometimes hear thresholds such as “use pgvector below ten million vectors”. Treat them as heuristics. Dimensions, filters, update rate, query concurrency, latency, recall target and the rest of your Postgres workload matter more than one round number.

### Memory and compression drive economics

A 768 dimensional float32 vector is about 3 KB before index and metadata overhead. Compression can therefore change the bill significantly.

- Scalar quantisation may store vector components at int8 instead of float32, reducing vector storage by roughly four times with a workload dependent recall cost.

- Binary quantisation can compress the vector representation far more aggressively, often paired with rescoring against higher precision vectors.

- Dimension truncation on models that support Matryoshka style embeddings reduces memory linearly and can stack with quantisation.

Those ratios describe the vector representation, not the total database footprint. Metadata, index structures, replicas, logs and backups still exist.

### Managed versus self hosted

Managed services trade infrastructure cost for engineering time and operational responsibility. Self hosting gives you more control and can make sense at sustained scale, inside an existing platform capability, or where residency, offline operation or sovereignty requirements dominate. There is no universal monthly bill at which the answer flips. Price the service, the people, reliability and the cost of failure together.

14 · Faithfulness

## Hallucination, factuality and faithfulness: measure the right failure

Being wrong about the world and being unfaithful to your sources are different failures with different fixes, and measuring the wrong one wastes the effort.

“What is the hallucination rate?” sounds like a precise question. It is not, unless you specify the task, dataset, model, scoring method and date. A grounded summary and an open factual answer are different problems and should not be compressed into one universal percentage.

### Two different questions

Factual correctness asks whether a claim is true in the world. Faithfulness or groundedness asks whether the claim is supported by the evidence supplied to the model. A response can be factually true but unsupported by the provided documents, or perfectly faithful to a source that is itself wrong.

Public benchmarks in 2025 and 2026 show large differences between open ended factual tasks and grounded summarisation tasks, but you should not divide those benchmark ranges and claim that grounding reduces hallucination by a fixed multiplier. They use different datasets, prompts and often different model sets.

The defensible conclusion is stronger anyway: grounding changes the failure mode. Instead of relying only on what the model recalls from its weights, you give it evidence that can be inspected and verified.

Current market

### Benchmark numbers move

Vectara replaced its earlier hallucination leaderboard in November 2025 with a harder benchmark based on more than 7,700 articles. In the published launch results the leading model was around 3.3% on that benchmark, while several reasoning variants exceeded 10%. Those are benchmark specific results, not promises about your application.

The Stanford AI Index 2026 also reports very high error or hallucination rates on some open factuality benchmarks across frontier models. Again, the useful lesson is not a single global number. It is that fluent open recall remains a poor substitute for evidence when the answer matters.

Reasoning can work against strict faithfulness
Some reasoning models perform worse than non reasoning counterparts on particular grounded summarisation benchmarks. That is plausible: the capability that helps a model infer, connect and extrapolate can conflict with an instruction to stay strictly inside the supplied source. Test reasoning on your own grounded task rather than assuming that a larger reasoning budget improves every metric.

### What to measure in your own system

- Faithfulness. Is each material claim supported by the evidence supplied?

- Citation correctness. Does the cited passage actually support the claim, not merely exist?

- Context precision. How much of the retrieved context was relevant?

- Context recall. Did retrieval supply enough evidence to answer fully?

- Answer relevance. Did the response address the question rather than an adjacent one?

- Abstention quality. Does the system decline to answer when evidence is insufficient, without refusing unnecessarily when evidence exists?

Frameworks such as RAGAS can help automate parts of this, and smaller faithfulness models can make continuous checks affordable. An LLM judge is still a model. Calibrate it against human labels on a sample, especially before using it for subjective or high consequence scoring.

MeasuredField note

### Zero observed fabricated citations

On a continuous market intelligence system we delivered for a life sciences instrumentation manufacturer, the acceptance criterion was unusually strict: every assertion had to trace to a source. The system tracked public funding across eight bodies, competitor activity and regulatory change across five territories, then produced a weekly executive digest. Final validation on the locked acceptance set produced 100% of items sourced and zero fabricated citations.

That does not prove hallucination is mathematically impossible. It proves that an operational system can achieve zero observed fabrication on a defined evaluation under a design that allows abstention. The result came from architecture rather than a mythical perfect model: source references travelled with claims, weak retrieval was rejected, and “insufficient evidence” was an acceptable outcome.

We used the same design principle on a separate technical assistant with a domain ontology: the system abstains rather than guesses.

### Rate the evidence, not the eloquence

| Rating | Status | Meaning | Rule |

| ★★★ | Confirmed | Primary source or internally verified evidence | Safe for external use under the normal review process |
| ★★ | Probable | Several consistent indirect signals, no direct confirmation | Internal use; verify before external communication |
| ★ | Inferred | Single weak source or logical inference | Verification required before it carries a decision |
| ○ | Unverified | No trustworthy evidence | Do not present externally as fact |

### Audit synthesis against the sources it compressed

When a model turns several documents into one executive summary, review the summary against the source set rather than reading the summary in isolation. We use five categories:

- Omission: important source material disappeared.

- Fabrication: a claim has no supporting source.

- Conflict: a figure, date or classification changed.

- Incorrect reconciliation: sources disagreed and the synthesis silently chose, averaged or invented a value.

- Scope violation: the synthesis moved beyond the agreed geography, entity set, period or brief.

On one anonymised engagement, that audit caught a market summary which attributed a revenue figure to the wrong stock exchange, presented two incompatible numbers and then recommended using one of them in a planning model. None of that authority existed in the source material. The prose was fluent enough that a normal read through would not have caught it.

Field note

### Fabrication caught at the gate, not at the source

On a market-entry engagement, a five-check audit — omissions, hallucinations, conflicts, reconciliations, scope — was run over a hundred and eighty-seven findings before anything reached the client. It caught more than thirty-five fabrications. The rate at generation was not zero; the rate at delivery was. That gap is the whole argument for a verification stage nobody is allowed to skip.

### What reduces operational fabrication

- Ground important answers in evidence or authoritative tools.

- Require source identifiers for material claims and verify citation correctness.

- Give the model an explicit abstention path.

- Derive relevance thresholds from labelled examples rather than intuition.

- Use deterministic checks wherever the output can be checked deterministically.

- Route low confidence or high consequence cases to human review.

Lower temperature can reduce variation, but it does not guarantee truth. The production objective is not “a model that never hallucinates”. It is a system with a measured residual error rate, visible provenance and controls that stop the residue from silently reaching a decision.

15 · Inputs

## Beyond text: multimodal, document AI and voice

Most document AI failures happen before the model sees anything, in the step that decided what the page said.

Engineering knowledge rarely arrives as neat prose. It sits in scans, drawings, datasheet tables, photographs, PDFs exported from spreadsheets, service notes and conversations. A text only pipeline can index the file while silently losing the information that matters.

Field note

### One source with no API

On a funding monitor covering eight public bodies, seven published an API and one did not. Rather than drop that body or scrape it blind, we built a verify-then-publish path for the single awkward source: extract, check against the published record, and only then let it into the digest. One source that behaves differently is normal. Designing as though they all behave the same is what breaks.

### Check what your pipeline actually reads

Before tuning embeddings, take twenty representative documents and inspect the extracted content. A scanned PDF may yield an empty string. A two dimensional table may become a sequence of numbers with the headings detached. A diagram may disappear completely. The ingestion job can report success while the knowledge base contains almost nothing useful.

If a material share of those twenty files produces gibberish, fix ingestion first. A better reranker cannot recover information that never entered the index.

### Three different technologies hide behind “document AI”

| Approach | What it does | Good fit |

| OCR | Turns image pixels into characters | Clean scans of ordinary text, at low cost and high throughput |
| Layout aware extraction | Preserves headings, columns, tables, key value pairs and spatial relationships | Invoices, forms, datasheets and repeatable layouts |
| Vision language models | Reason over the page or image itself | Irregular layouts, screenshots, charts, mixed content and difficult visual context |

Do not treat that table as a quality ladder. The most general model is not automatically the best production choice. A stable invoice format may be cheaper and more reliable with a specialised layout extractor plus deterministic validation than with a large vision model.

[Figure] Preserve where every value came from. OCR turns pixels into text; layout-aware models recover structure; multimodal models interpret richer content. Consequential extraction should carry page, cell or region provenance into review.

### Tables deserve their own design

A table is not a paragraph with extra spacing. Meaning lives at the intersection of row and column. Flatten “500” away from “maximum operating pressure”, “model B” and “bar” and the value becomes useless.

Good pipelines preserve the structured table, create a searchable description for semantic discovery, and use a structured query when the user needs a specific cell or calculation. This is the same principle as giving a language model SQL access instead of asking it to reason over a screenshot of a database.

### Drawings and photographs

Vision language models can already extract useful information from engineering drawings: title blocks, revision identifiers, notes and obvious annotations. That alone can make a drawing archive searchable. Reliable interpretation of complex geometric tolerancing or safety critical dimensions is a different standard and should be validated against a purpose built evaluation set before anyone claims production readiness.

Field photographs are often a more immediate opportunity. Read the serial number from a nameplate, identify the product family, retrieve service history and present the technician with the right manual. Each step is bounded and measurable.

Current market

### Voice is becoming another business interface

Real time speech recognition, speech synthesis and low latency multimodal models have made voice much more practical. The interesting use case is not a talking chatbot for its own sake. It is capturing information where typing is inconvenient: a service engineer walking a site, an operator with gloved hands, a maintenance handover or a customer support call.

For companies whose highest value knowledge sits with a handful of experienced engineers, recorded and transcribed conversations can become part of a deliberate knowledge capture programme. Treat consent, retention and access controls exactly as you would any other business record.

[Figure] Voice does not remove the harness. The audio channel changes turn-taking and latency, but tool permissions, approvals, retrieval, logging and evaluation all stay in the surrounding system.

### Evaluate extraction field by field

Create a labelled set of representative documents and record the correct values. Measure field level accuracy, then measure a second number that is often more important: how frequently does the system invent a value when the field is actually blank?

For consequential extraction, return provenance with every field. Page number plus a bounding region, cell reference or source span lets a reviewer verify the result quickly and makes invented values much easier to spot.

16 · Layer three

## Tools, MCP and the harness

A tool call is a permissioned software operation, and the layer that grants or refuses that permission is where safety actually lives.

Text generation becomes operational AI when the model can request actions: search a database, run code, query an ERP system, create a ticket or prepare an email for approval. The important engineering lives between the model's request and the real world.

### Tool use, mechanically

Your application tells the model which tools exist, what each one does and which arguments it accepts. The model can then emit a structured request such as call get_invoice with id=INV-4471. The host application decides whether that request is valid, executes the function if permitted, and returns the result to the model.

The model itself does not magically acquire database credentials because it generated the tool call. Execution belongs to the surrounding runtime. That separation is where authentication, authorisation, validation and human approval belong.

[Figure] Tool calling is a permissioned software operation. MCP standardises a connection pattern, but it does not replace authentication, least privilege, network controls, validation or human approval for consequential actions.

### MCP: a common interface for tools and context

The Model Context Protocol, introduced by Anthropic in 2024 and now under the Linux Foundation's Agentic AI Foundation, standardises how compatible clients discover and use external tools and resources. Adoption across model providers, developer tools and enterprise software has made it one of the main interoperability standards in the agent ecosystem.

A2A, originally developed by Google and also moved into Linux Foundation governance, tackles a different boundary: agents discovering and communicating with other agents. The shorthand is useful: MCP connects an AI client to tools and data; A2A addresses agent to agent interoperability.

Measured

MCP does not remove ordinary security requirements
A 2026 measurement study identified 7,973 live remote MCP servers and found 40.55% exposing tools without authentication. Among the subset of testable OAuth enabled servers, researchers also found widespread implementation flaws. The lesson is not that MCP is inherently unsafe. It is that rapid adoption does not excuse authentication, network scoping, least privilege and server provenance.

### The harness

The harness is the operating environment around the model: tools, credentials, sandbox, permissions, validators, budgets, logging and rules governing what can execute.

Think of a capable contractor arriving at a factory. The model is the contractor. The harness is the induction, badge, locked doors, permit to work, test equipment and sign off process. Capability matters, but the controls determine what you can safely let that capability touch.

### Production rules that age well

- Prefer a small, distinct toolset. Overlapping tools create selection ambiguity and every description consumes context.

- Return the information needed for the next decision. A 300 line API response is rarely useful context. Transform it into the few fields the model needs.

- Apply least privilege per tool. Read access by default, narrow write scope, explicit approval for high consequence or irreversible actions.

- Sandbox generated code. Isolate execution, control network egress and do not mount secrets the task does not require.

- Make failures explicit. “No invoice found for INV-4471” is a useful tool result. An empty or ambiguous response invites the model to fill the gap.

- Verify the server or connector itself. Tool descriptions and MCP packages are part of your software supply chain. Pin versions, review permissions and monitor changes.

17 · Reliability

## Structured outputs: when prose has to become dependable data

Valid JSON is not valid business data, so constrain the shape with a schema and validate the meaning separately.

A human can forgive an extra sentence. An API cannot. The moment model output feeds software rather than a reader, structure becomes an engineering contract.

### Three levels of control

| Method | What you get | What can still go wrong |

| Prompt only | “Return JSON with these fields” | Extra prose, missing keys, code fences, invented fields, invalid syntax |
| JSON mode | Syntactically valid JSON | The schema, types or values can still be wrong |
| Schema constrained output | Generation restricted to a declared structure | Values can still be semantically incorrect |

Schema constrained decoding can prevent invalid field names or types because impossible continuations are excluded while the model generates. That is a major improvement over “please format this correctly”. It does not make the extracted invoice date true.

### Design schemas for both the model and the workflow

- Keep structures shallow where possible. Deep nesting increases the number of relationships the model must maintain.

- Use enumerations for closed choices. They remove ambiguity and simplify downstream handling.

- Include an explicit missing value state. “Not found” is safer than forcing the model to populate every field.

- Return evidence. Source span, page, record ID or quotation lets the next system or reviewer verify the value.

- Name fields semantically. net_amount_gbp provides more context than value_3.

Field note

### Four states, not pass or fail

The same engagement classified every finished output into one of four states before hand-over: deploy now, deploy after verification, template only, or blocked by an NDA constraint. Thirty-nine of sixty-two items cleared immediately; the rest were routed rather than rejected. A binary valid or invalid would have discarded most of the work.

### Generate, validate, repair, escalate

- Generate against the schema.

- Validate structure and deterministic business rules. Does the total equal the line items? Is the part number in the catalogue? Is the date plausible?

- Repair by sending the rejected output and precise validation error back for one or more bounded correction attempts.

- Escalate when the retry budget is exhausted or the consequence exceeds the automation threshold.

[Figure] Valid JSON is not the same as valid business data. Use schema constraints to control form, deterministic checks to control rules, bounded repair for recoverable errors, and an explicit exception path for everything else.

An exception queue is evidence of a controlled system, not evidence of failure. A workflow that always returns something will eventually return something wrong without telling you.

### Ordinary integration discipline still applies

Use idempotency so a retry does not double post an order. Add timeouts and circuit breakers. Keep versioned prompts and schemas in source control. Run contract tests against a golden set after model or prompt changes. If you have built industrial integrations before, none of this should feel exotic. The upstream component is probabilistic; the surrounding software should not be.

18 · Layer four

## Agents: the model gets another turn

An agent is a model given another turn, plus the software discipline to survive taking it.

The useful definition of an agent is simple: a model can use tools, observe what happened and decide what to do next. The defining feature is the loop, not the branding.

### Workflow versus agent

If your code determines the sequence and the model fills in individual steps, you have an AI enabled workflow. That is often exactly what you want: predictable, testable and cheap. If the model chooses the next action based on what it has just discovered, you have an agentic loop.

Most agent systems implement some variation of:

gather context → decide → act → observe → verify → repeat

[Figure] The verification step is load-bearing. Where possible, success should be checked against something the working model cannot simply declare true: tests, schemas, database state, explicit criteria or an independent verifier.

### Three questions separate a demo from a production agent

How does it know the task is complete?
Do not rely only on the working model asking itself whether it has finished. Use an external success condition where possible: tests, a schema, a database state, an explicit checklist or a separate verifier with no permission to change the result.

What stops it?
Set ceilings on steps, wall clock time, tokens, tool calls and spend. An agent that cannot make progress should fail into a visible state, not continue until the budget disappears.

Where does progress live?
Long running work needs durable state outside the model context. Save completed steps, decisions, artefacts, errors and the next intended action so a new process, model session or human can continue after a crash or context reset.

### Durable execution: the missing production layer

A serious agent should survive ordinary software failure. That means checkpoints, idempotent actions, queues, leases or locks where necessary, bounded retries and clear recovery semantics.

- Checkpoint before and after consequential actions. You need to know whether the purchase order was created before retrying the step.

- Make writes idempotent. Replaying the same request should not create the same invoice twice.

- Use compensating actions for reversible workflows. If step four fails after step three committed, know how to unwind or escalate.

- Persist a compact task state. Do not depend on a 200,000 token transcript as your only record of what happened.

- Provide a kill switch. Operators need a way to stop new actions while preserving evidence for diagnosis.

[Figure] An agent should survive an ordinary software failure. Checkpoints, idempotent actions and durable state let a new process or model session continue without guessing what happened before the crash.

The best analogy is shift work. A fresh engineer should be able to take over from the previous shift using the written state, not by inheriting the previous person's brain.

### When not to build an agent

| Task shape | Build | Reason |

| One bounded step | Single model call or deterministic function | Anything else is overhead |
| Several known steps in a known order | Workflow | Predictable, testable and easy to audit |
| Known branches based on content | Router plus workflows | Keep determinism where the problem allows it |
| The next step genuinely depends on what is discovered | Agent | Flexibility earns its cost only here |

Agentic architecture adds cost, latency and debugging difficulty. Use it to buy flexibility you actually need, not because “agent” sounds more advanced than “workflow”.

19 · Layer four, extended

## Autonomy, multi agent systems and orchestration

Autonomy is not a property of the model but a permission you grant, and the ceiling should be set by blast radius.

Autonomy is not a yes or no property. The useful questions are: what may the system decide, what may it execute, what limits apply, and who notices when the outcome is wrong?

Field note

### Unattended pipeline, attended send

A weekly intelligence system we operate runs its entire collection and synthesis pass unattended on a Sunday, then stops. A staging preview goes to one named person, who replies to approve, and only then does the Monday 08:00 delivery leave. The expensive work is autonomous. The irreversible step is not.

### An autonomy ladder

| Level | System behaviour | Typical use |

| Suggests | Produces analysis or a draft; a person decides and acts | Early deployment, customer facing, legal, financial or novel work |
| Acts with approval | Prepares a specific action; a person authorises it | A strong default for consequential business workflows |
| Acts within limits | Executes inside a bounded policy and escalates exceptions | High volume, reversible, well measured work |
| Acts freely | Executes with only retrospective review or none | Sandboxed, low consequence, recoverable tasks |

Choose the level with a blast radius test: what is the worst credible outcome, how quickly can you detect it, and can you reverse it? Model capability does not answer those questions.

[Figure] Set autonomy from consequence, not model capability. Drafting an internal note and issuing a refund may use the same model, but the acceptable control level is different because the blast radius is different.

### Multi agent systems

A coordinator can split work across several workers with different tools or context. The biggest benefit is often context isolation rather than mythical specialisation. Five researchers can each read twenty documents and return concise findings, allowing the coordinator to work with five summaries rather than a hundred full documents.

The cost is multiplication: more tokens, more latency, duplicated research, inconsistent conclusions and harder tracing. Start with one agent and a well designed toolset. Add multiple agents when task decomposition or context isolation produces a measured gain.

Rule of thumb

### Computer use agents

Some agents can operate a graphical interface through screenshots, mouse and keyboard events. This is useful when an application has no suitable API, but it is a fragile integration surface. Buttons move, labels change, pop ups appear and visual state can be ambiguous.

API first, GUI second. When computer use is unavoidable, verify state after every consequential action, keep permissions narrow and require approval before irreversible steps. Treat the interface as untrusted external state, not as a deterministic function call.

[Figure] A screen is not an API. An interface can drift, carry misleading text, or fail to show clearly whether the action worked. The loop has to observe the new state rather than assume the click succeeded.

### Orchestration: the plumbing behind the intelligence

- Routing. Send routine traffic to an inexpensive model and reserve frontier capability for cases that need it. The saving depends on your workload; measure quality before and after routing rather than assuming a percentage.

- State. Persist progress so a process restart does not erase an hour of work.

- Retries and fallback. Handle rate limits, transient tool errors and provider outages without duplicating side effects.

- Observability. Trace model calls, tool calls, retrieval, decisions, latency, token use, cost and final outcome.

- Caching. Reuse static prompt prefixes and repeated intermediate results where providers or your own architecture allow it.

- Evaluation. Record whether the task was completed correctly, not merely whether every API call returned 200.

LangGraph, vendor SDKs, workflow engines such as n8n and ordinary application code can all implement these patterns. Framework choice changes faster than the underlying requirements.

20 · The market

## Choosing a model without chasing every release

Choose for the task, the deployment route and the cost of leaving; the leaderboard is the least durable of the four criteria.

Model names age quickly. Workload categories age much more slowly. Choose the smallest, cheapest model that clears a measured quality bar for the task, and keep the surrounding system portable enough to change later.

### Four useful tiers

| Tier | Use it for | Typical trade |

| Frontier | Hard reasoning, complex code, long agent runs, high consequence analysis | Highest capability, usually highest token cost and latency |
| Mid tier | RAG, summarisation, drafting, extraction, ordinary business analysis | Often the best balance of capability, speed and cost |
| Small / fast | Classification, routing, tagging, moderation, simple extraction | Low latency and low cost, sometimes local or on device |
| Specialist | Embeddings, reranking, speech, vision, image generation and other narrow jobs | Purpose built models often beat a general model on their own task |

A mature application can use several tiers in one request. A small router may classify the task, a specialist retriever finds evidence and a stronger model handles the difficult final synthesis. Sending every call to the largest model is easy architecture, not necessarily good architecture.

### Closed versus open weight

Closed models are accessed through a provider or cloud platform. You avoid model serving, gain enterprise contracts and usually receive the latest frontier capability. In return you accept provider dependency, usage pricing and whatever data handling terms you negotiate.

Open weight models make the trained parameters downloadable under a licence. You can run them yourself, rent infrastructure or call a third party host. That creates deployment choice, but “open weight” does not automatically mean open source, unrestricted or self hosted.

Open weight is a deployment property, not a hosting decision
You can consume open weights through a serverless API and never operate a GPU. You can also run the same weights entirely offline. Those routes have very different economics and sovereignty properties even though the underlying model file is identical.

### Do not reduce the decision to benchmark rank

Open weight capability has moved rapidly towards the closed frontier during 2026, particularly on coding, reasoning and agentic benchmarks. The gap is workload dependent and changes too quickly to summarise as a permanent number of months. On extraction, classification, grounded RAG and structured drafting, open models can already be entirely adequate. On long horizon tool use, the best closed models can still justify their premium when reliability compounds over many steps.

### Fine tuning is one tool, not the default answer

Fine tuning can teach a repeatable behaviour, format, domain style or specialised classification pattern when you have good examples. It is usually a poor way to keep changing business facts current. For knowledge, prefer retrieval or direct tools. For behaviour, consider fine tuning after you have proved that prompting and examples are insufficient.

### A model selection process that survives the market

- Collect twenty to fifty representative tasks from the real workload, then expand the set as the product matures.

- Define what an acceptable answer looks like before seeing candidate outputs.

- Test models across more than one tier and, where practical, more than one provider.

- Score correctness and task success first, then cost, latency and failure behaviour.

- Choose the cheapest option that clears the quality threshold for that task.

- Keep prompts, tool contracts and evaluations versioned so a model swap is an engineering change, not a leap of faith.

21 · Market structure

## The open weight surge, and why it matters for sovereign AI

Open weights are a property of the model rather than a hosting decision, and they are what makes a sovereign deployment possible at all.

The strategic change in 2026 is not simply that open weight models became cheaper. Several Chinese laboratories now publish models close enough to the frontier that European and UK businesses can make a genuine choice between capability, price, control and operational independence.

### The market moved quickly

DeepSeek, Alibaba's Qwen team, Moonshot AI, Zhipu AI and MiniMax all pushed large open or openly deployable model families forward during 2026. Moonshot's Kimi K3 paper describes a 2.8 trillion parameter mixture of experts model with 104 billion active parameters, native vision and a one million token context window. Other families have competed through permissive licences, aggressive prices and strong coding or agent benchmarks.

Do not turn that movement into a permanent leaderboard sentence. Model releases, hosted prices and licences can change within weeks. Use this section to understand the economics, then verify the live table before procurement.

### Why publish the weights?

Several incentives line up without requiring altruism.

- Commoditise a complement. A company that sells cloud, compute, chips or enterprise integration can benefit from making the model layer abundant.

- Buy distribution. Published weights spread through Hugging Face, developer tools, fine tunes and third party hosting in a way a less trusted foreign API may not.

- Compete through efficiency. Constraints on access to leading accelerators have increased the incentive to improve sparsity, precision, attention efficiency and serving software. The causal effect of export controls on capability remains debated; the engineering focus on efficiency is visible.

- Shape standards and defaults. Models that developers build around influence APIs, deployment stacks and ecosystem habits.

- Support a wider industrial strategy. State policy, cloud economics and model economics are not independent in the Chinese AI market.

The continuity lesson from June 2026
On 12 June 2026 the US government imposed export controls on Anthropic's Fable 5 and Mythos 5, forcing Anthropic to suspend access while it could not reliably verify user nationality. The controls were lifted on 30 June and Fable 5 returned globally on 1 July. The interruption lasted only weeks, but it demonstrated a real dependency: access to a closed frontier service can change because of legal or geopolitical decisions outside your contract.

Downloaded weights running on infrastructure you control cannot be remotely withdrawn in the same way. That does not mean every company should self host. It means a load bearing AI dependency deserves the same continuity analysis as any other critical supplier, including a tested fallback where the consequence justifies it.

### Mixture of experts: the model can be huge without using all of itself

A dense model uses essentially all of its model parameters for each generated token. A mixture of experts, or MoE, routes each token through only part of the network. Think of a hospital with hundreds of specialists on staff but only a few consulted for a particular patient.

This creates two numbers:

- Total parameters mainly determine how much memory is required to hold the model weights.

- Active parameters strongly influence how much compute each token requires, although routing, attention, memory bandwidth and serving implementation also matter.

Kimi K3, for example, reports 2.8T total parameters and 104B activated. That is why a model can have enormous storage requirements yet an inference profile closer to a much smaller dense model than the headline parameter count suggests.

[Figure] Total and active parameters answer different questions. The full expert pool drives storage and VRAM requirements; the routed subset drives much of the per-token compute. That is why an MoE can be enormous on disk yet relatively efficient to serve.

### Low prices come from both engineering and market structure

MoE sparsity, lower precision arithmetic, efficient attention and better serving stacks all reduce compute per token. Third party hosts then compete to serve the same downloadable weights, which can push hosted prices down further. A self hosted copy does not automatically beat those economics because your GPU may sit idle while a specialist provider pools demand across customers.

Current market

### A dated market snapshot, checked 20 August 2026

This table is orientation, not a permanent price list. Prices refer to the named hosting route and must be rechecked before use in a proposal.

| Model / family | Access | Useful fact | Illustrative hosted price |

| DeepSeek V4 Flash | Open weight plus DeepSeek API | 1M context; aggressive low cost tier | DeepSeek direct from 17 Aug: $0.22 / $0.66 off peak, $0.44 / $1.32 peak per 1M cache miss input / output |
| Kimi K3 | Open weight under its published licence | 2.8T total / 104B active MoE, native vision, 1M context | Varies materially by host |
| GLM 5.x family | Open weight releases with permissive options in the family | Strong coding and agent focus | Varies by model and host |
| Qwen family | Open weight and hosted variants | Large developer ecosystem and broad modality coverage | Varies by endpoint |
| Claude Fable 5 | Closed API | Frontier tier; June 2026 continuity event noted above | $10 / $50 per 1M input / output at launch |

Why the DeepSeek line changed between drafts of this guide
V4 Flash launched with $0.14 cache miss input and $0.28 output pricing per million tokens. DeepSeek introduced peak and off peak pricing effective 17 August 2026, only days before publication of this guide. That is exactly why every time sensitive price should carry a provider and a date.

### Open weight does not settle the governance question

Separate where the weights came from from where your data goes. Calling a hosted API means prompts and outputs are processed by that provider under its contractual and jurisdictional framework. Downloading the same weights and running them inside infrastructure you control does not transmit inference data to the model creator merely because of the model's nationality.

Self hosting still raises other questions: licence terms, supply chain provenance, malicious or accidental behaviour, vulnerability management and model evaluation. Record the model, version, licence, source, checksum and owner in your AI inventory so provenance is auditable.

22 · Evidence

## Benchmarks: useful exams, poor procurement decisions

A benchmark is a useful exam and a poor procurement document, because it measures a task that is not yours.

Benchmarks help us compare model capability under controlled tasks. They become misleading when a leaderboard score is treated as a forecast for your production workflow.

| Benchmark family | What it tries to test | How to read it |

| MMLU Pro, GPQA | Broad and expert academic knowledge | Useful for general capability; some top benchmarks are increasingly saturated |
| SWE bench | Resolving real software issues | Useful coding signal, highly sensitive to harness, tools and evaluation quality |
| Terminal Bench | Multi step command line tasks | Closer to tool using agent work than static question answering |
| GAIA family | General assistant tasks using tools and information | Measures the system as much as the base model when scaffolding is allowed |
| τ bench | Business dialogues under explicit rules | Useful because it exposes policy following and tool use rather than trivia |
| ARC AGI 2 | Novel visual reasoning and generalisation | Interesting capability signal with deliberately unfamiliar tasks |
| OSWorld, WebArena | Operating computers and websites | Relevant to computer use agents, but the environments themselves drift |

### Why leaderboard scores need context

- Contamination. Public tasks can appear in training or derivative datasets. Evaluation suites themselves also contain flawed or ambiguous items.

- Scaffolding. A model with a strong tool harness, retry strategy and context setup can outperform the same model in a bare run. Sometimes the system is what you are benchmarking.

- Saturation. When top models cluster near the ceiling, small score differences carry little information.

- Methodology. Single attempt, best of several attempts, reasoning budgets and tool access are not directly comparable.

- Production drift. Benchmarks are snapshots. Your applications interact with changing data, APIs, users and permissions.

Field note

### The baseline was a person, not a benchmark

For a support triage system, the numbers that mattered were routing accuracy above ninety per cent and time-to-route under two minutes, measured against a managing director who had been spending over thirty minutes triaging each ticket himself. No model leaderboard would have told us whether that was working.

### Evaluate the finished task, not only the model

Production evaluation should extend beyond RAG. Track the outcome that matters to the workflow:

- Task success: did the requested job finish correctly?

- Deterministic checks: did tests, business rules or calculations pass?

- Human correction rate: how often did a person have to change the result before use?

- Factuality and faithfulness: are material claims true and supported?

- Latency and cost per completed task: a cheaper token can produce a more expensive workflow if it retries more often.

- Safety and permission failures: did the agent attempt something outside its allowed scope?

- Regression by version: model, prompt, tool and retrieval changes should all be traceable to evaluation results.

Use public benchmarks to shortlist, then use your own work to decide
A public leaderboard can tell you which models deserve testing. Your own evaluation set tells you which one deserves production. Start with real tasks and accepted answers, then keep adding difficult and failed cases. That dataset becomes one of the most durable assets in an applied AI programme.

23 · Buying it

## What an API call costs, and what an AI “wrapper” really is

You are billed for the request the model sees, not the message the user typed, and the gap between the two is where the bill comes from.

Most AI products eventually reduce to software sending context to a model service and receiving output. Understanding that transaction makes pricing, architecture and vendor differentiation much easier to judge.

### The transaction

An API lets one software system request a service from another. A model request typically specifies the model, instructions or messages, tool definitions, output constraints and generation settings. The provider returns generated content, tool calls or structured data together with usage information.

The exact state model depends on the API. Some calls are completely self contained. Some provider products can store conversations, hosted files, cached prefixes or other server side state. The durable principle is the one from context engineering: the model can reason only over what the current inference makes available, however the surrounding platform assembled it.

# Simplified example
POST /v1/responses
{
"model": "example-model",
"input": "Summarise this contract: ...",
"max_output_tokens": 1000
}

# Typical usage metadata
{
"input_tokens": 8412,
"output_tokens": 316
}

### Tokens are the basic meter

Providers usually price input and output separately per million tokens. Output often costs several times more than ordinary input because generation is more computationally expensive than processing an existing prefix. Reasoning tokens, cached input and batch discounts can change the effective ratio significantly.

A million English text tokens is roughly three quarters of a million words as a very loose orientation. A typical business RAG request might consume thousands or tens of thousands of input tokens and hundreds of output tokens. Individual calls can therefore feel cheap while aggregate spend grows quickly through volume, retries, tools and agent loops.

Current market

### A dated price orientation, not a quotation

| Category | Typical 2026 shape | What to remember |

| Frontier closed | Several dollars input and tens of dollars output per million tokens | Pay for difficult reasoning where it improves task success |
| Mid tier closed | Lower single digit input, roughly low double digit output | Often the general production workhorse |
| Hosted open weight | Can range below $1 output to several dollars depending on model and host | Same weights may have very different provider prices |
| DeepSeek V4 Flash direct | From 17 Aug 2026: $0.22 / $0.66 off peak and $0.44 / $1.32 peak for cache miss input / output | Time of day and cache status now affect the direct API bill |

Always check the live provider page before budgeting. This guide itself had to change a DeepSeek price between drafting and publication because the vendor repriced the API on 17 August 2026.

### Prompt caching, batching and routing change the economics

Repeated static prompt prefixes can often be cached at a steeply reduced input price. Non urgent batch endpoints can offer substantial discounts. Model routing can move simple calls to a cheaper tier. The right unit is therefore not “cost per million tokens” but cost per successfully completed business task.

### What is a wrapper?

At the thin end, a wrapper is little more than a user interface, a prompt and a model API call. The label becomes misleading when the surrounding product performs meaningful integration, workflow, retrieval, permissions, audit, data processing or evaluation.

### The substitution test

Ask: could a competent user get roughly 80% of the value by pasting a good prompt into a general assistant? If yes, the differentiation is thin. If the product depends on proprietary data, deep integration, multi step execution, compliance controls or an operational feedback loop, the model call is only one component.

### Why thin wrappers face pressure

AI native products often carry lower gross margins than conventional software because inference remains a variable cost. Industry analyses commonly place some AI application businesses around 50 to 60% gross margin compared with roughly 70 to 90% for traditional software businesses, although the spread is wide and architecture matters. We have removed a previously cited “3.2× funding” figure from this guide because we could not verify the primary evidence well enough to justify the precision.

The deeper strategic risk is substitutability. If the only proprietary asset is a prompt, foundation model providers can add the feature and competitors can reproduce it quickly. Defensibility comes from data, workflow, integration, distribution, trust and accumulated feedback.

### Questions worth asking a vendor

- What happens if the underlying model is repriced, deprecated or temporarily unavailable? Look for portability and a tested evaluation set.

- What do you provide that I cannot reproduce in a general model interface? Data, workflow, integration and control are credible answers.

- Show me the evaluation methodology. Not only the demo.

- Where is our data processed and what does the contract say?

- What is the cost per completed task at our expected volume?

### Internal wrappers can be exactly the right solution

An internal tool does not need a venture capital moat. It needs to save time, improve quality or reduce risk. Many successful corporate AI applications are technically wrappers around models, retrieval and company systems. That is not a criticism.

Field note

### Where the value actually came from

On an Ascentis AI engagement with a battery safety calorimetry manufacturer, a private AI sales adviser moved from MVP to production in eight weeks, with an estimated 5× return and payback in around two and a half months. On a thin film deposition manufacturer's support operation, automated triage moved from about thirty minutes to under two minutes per case at more than 90% accuracy.

The model API was not the hard part. The first result depended on confidential battle cards and pricing strategy. The second depended on process redesign, domain knowledge and integration. That is the practical line between “an API wrapper” and a system the business depends on.

A useful analogy
The model is electricity. Electricity is a commodity; the factory that runs on it is not. The value lies in what the model is connected to, what the workflow knows, what it can safely do and how difficult the capability would be to remove.

Related on this siteWhether to buy a tool or build one: Off-the-shelf vs bespoke AI. To put your own figures against the arithmetic above, the AI ROI calculator.

24 · Deployment

## Cloud API, hosted open weight, self hosted or on device

Four different things get called “self-hosted”, and which one you need is usually decided by a contract rather than by a benchmark.

Deployment is not a moral choice between cloud and local. It is a trade between capability, economics, control, sovereignty, latency and operating burden. The mistake is to collapse all of those into the single question “is our data sensitive?”

| | Frontier API | Hosted open weight | Self hosted | On device |

| What it is | Call a closed model service | Call downloadable models on somebody else's infrastructure | Run model weights on infrastructure you control | Run a smaller model on the endpoint |
| Setup | Fast | Fast | Material engineering and operations | Application and hardware integration |
| Capability | Latest frontier | Strong and rapidly improving | Same open weights, limited by your hardware and serving stack | Smaller models and specialist workloads |
| Cost shape | Variable per use | Variable per use, often lower | Capacity and people cost whether busy or idle | Hardware already distributed across devices |
| Data path | Processed by provider under contract | Processed by host under contract | Can remain inside your controlled environment | Can remain on the endpoint |

### Four words that should not be used interchangeably

- Data residency: where data is physically processed or stored.

- Localisation: a policy or legal requirement that data remain in a defined territory or environment.

- Jurisdiction: which legal regimes and government access powers can reach the provider or data.

- Sovereignty: how far you can operate, govern and continue the critical AI capability under controls you own rather than controls a supplier or foreign government can change.

A workload can have EU residency without being sovereign. It can also be sovereign without being physically inside one office, for example when an organisation controls the model, keys, deployment and operating environment in a European datacentre under an appropriate legal structure. Define the requirement before choosing the hosting label.

[Figure] Define the requirement before choosing the hosting label. The economic case for self-hosting is mainly utilisation. The strategic case may instead be operational control, continuity, disconnected operation or sovereignty.

Rule of thumb

### The economics of self hosting

A GPU cost divided by its theoretical maximum throughput can make self hosting look spectacularly cheap. That calculation usually assumes high utilisation, ideal batches and no engineering cost. Business traffic is rarely so obliging.

The honest comparison is often not self hosted open weight versus frontier closed API. It is self hosted open weight versus a specialist provider serving the same open model. At low or variable utilisation, shared infrastructure frequently wins because the provider keeps the hardware busy across many customers.

People also belong in the calculation: drivers, inference software, upgrades, monitoring, security, capacity planning, backups, on call support and evaluation after model changes.

The distinction we use with clients
The economic case for self hosting is mainly a utilisation question. The strategic case can be a sovereignty question. Offline operation, client mandates, egress restrictions, operational independence, continuity, export controls or the need to control logs and weights can justify self hosting even when an API is cheaper.

Field note

### The deciding constraint is often contractual, not technical

Two of our self-hosted builds were not chosen for latency or unit cost. On a sales adviser, the corpus was confidential battle cards, competitor analysis and pricing rationale that could not transit a third-party API at all. On a market intelligence system, synthesis runs on a UK-resident host so that no third-party model API sits in the production data path. In both, a queryable prompt and response audit trail was part of the requirement rather than a nicety.

### When self hosting is usually justified

- Sustained, predictable utilisation where your measured total cost beats hosted alternatives.

- Air gapped or disconnected operation.

- Contractual or security rules that prohibit the required data from reaching an external processor.

- A sovereign operating requirement, including control of model availability and deployment lifecycle.

- Heavy customisation or integration with specialised hardware where a generic API is a poor fit.

### On device inference

Phones, laptops, vehicles and industrial endpoints increasingly include neural accelerators capable of useful local inference. On device models are attractive where there is no network, where round trip latency is unacceptable, where raw data should not move, or where thousands of modest local inferences would be inefficient to centralise.

The mature architecture is often hybrid: classify, transcribe or filter locally; escalate difficult reasoning to a larger hosted or central model only when required.

Rule of thumb

### Sovereign AI includes the software supply chain

Running weights locally is not the end of the sovereignty conversation. Record where the weights came from, the licence, checksum and version. Pin container images and Python packages. Scan dependencies and model artefacts. Control who can update the inference server. Test the new version before promotion. If a supposedly sovereign system downloads arbitrary models or packages from the internet at runtime, its operating boundary is not as sovereign as the network diagram suggests.

[Figure] Treat the model as a critical software artefact. Import it through a controlled path, know exactly what is deployed, prevent silent change, and be able to prove the system can keep running inside the boundary you claim for it.

Related on this siteThis guide covers the concepts. For the numbers and the build — a working reference specification, real UK prices and the break-even arithmetic — see the Self-Hosted AI Guide. For the side-by-side comparison, Self-Hosted vs Cloud AI. For a decision walkthrough on your own workload, Where Should Your AI Live.

25 · Infrastructure

## GPUs, quantisation, inference servers and Docker

What fits on a card is set by weight precision and the KV cache, and the cache is the part that grows while you are using it.

You do not need to become a GPU engineer to buy AI well. You do need enough vocabulary to challenge optimistic capacity numbers and understand why a model that runs on a developer laptop is not automatically ready for production traffic.

### Why GPUs dominate language model inference

Modern neural networks perform enormous amounts of parallel matrix arithmetic. GPUs contain many execution units and very high memory bandwidth, which makes them far better suited to this workload than general purpose CPUs for large models.

The first constraint is often VRAM, the memory attached to the accelerator. Model weights, KV cache, temporary activations and serving overhead all compete for it.

### Model weights, approximately

Quantisation is a photograph saved at a lower colour depth. Much smaller, and you cannot see the difference — until the point where you suddenly can.

An 8B model contains about eight billion learned parameters. As a first approximation, weight memory equals parameter count multiplied by the storage precision.

| Representation | Approx. bytes per parameter | 8B weight memory | Typical trade |

| FP16 / BF16 | 2 | ~16 GB | High fidelity reference serving |
| FP8 | ~1 | ~8 GB | Lower memory with hardware and model support |
| 4 bit quantised | Roughly 0.5 to 0.7 including quantisation metadata | ~4 to 6 GB | Large memory saving with task dependent quality loss |

[Figure] Weight precision is the first memory lever. Four-bit quantisation can cut weight memory dramatically, but quality loss is model- and task-dependent. The deployed footprint is larger than the bars because KV cache, runtime buffers and serving overhead also consume VRAM.

Quantisation reduces numerical precision so the same weights occupy less memory and may run faster. The quality cost depends on model, method and task. “Four bit is indistinguishable” is not a universal fact; evaluate the quantised build you plan to deploy.

### The KV cache is the memory that grows with use

The KV cache is the conversation’s running notes. Cheap to add a line to, never rewritten from scratch, and the reason the twentieth turn of a long chat costs more than the first.

During autoregressive generation, the server stores attention keys and values for the sequence so it does not recompute the whole prefix at every token. This KV cache grows with context length, batch size and concurrent sequences. Long context and many simultaneous users can therefore consume more memory than a single user test suggests.

### Local runners versus production serving

Ollama and llama.cpp make local experimentation remarkably easy and support CPU, consumer GPU and Apple Silicon configurations. Ollama can process parallel requests subject to configuration and available memory; it should not be described as inherently sequential.

vLLM, SGLang, TensorRT LLM and similar servers are designed around high throughput datacentre serving. Features such as continuous batching, efficient KV cache management, tensor parallelism and scheduling make the difference under concurrent load.

Continuous batching
Requests finish at different times. Continuous batching lets the server reuse freed capacity immediately rather than waiting for an entire fixed batch to complete. This increases accelerator utilisation and reduces the penalty created by one unusually long response.

Paged KV cache management
Serving systems can allocate KV cache in blocks rather than reserving large contiguous regions per request. The idea resembles virtual memory: use memory where it is needed and reduce fragmentation. The exact implementation differs between servers, but the operational benefit is higher useful concurrency from the same VRAM.

| | Local / developer runners | Production inference servers |

| Primary objective | Easy local execution and broad hardware support | Throughput, batching, monitoring and accelerator efficiency |
| Concurrency | Supported, but bounded by local memory and configuration | Designed to schedule many concurrent sequences efficiently |
| Hardware | CPU, consumer GPUs, Apple Silicon, local workstations | Commonly datacentre accelerators and multi GPU systems |
| Best fit | Evaluation, development, edge and modest local services | Customer facing or high volume inference |

### Docker has three different jobs here

- Reproducibility. Pin CUDA libraries, Python packages and the serving stack so test and production environments match.

- Isolation. Run generated code in disposable containers with controlled credentials, filesystem and network access.

- Deployment. Package services into units that ordinary infrastructure tooling can scale and update.

Kubernetes is useful when you genuinely have the scale and operational capability to justify it. It is not a maturity badge. Docker Compose or ordinary managed container services are often the better answer for an SME deployment.

### How to interrogate a throughput claim

Ask for the model, quantisation, hardware, batch size, input length, output length, concurrency, time to first token, steady state tokens per second and assumed utilisation. A number measured on short prompts with full batches does not predict a workload built around 100,000 token contexts and bursty office hours.

26 · Platform

## Hosting and scaling an AI application

An AI application is an ordinary distributed system with one unusually slow, unusually expensive dependency in the middle of it.

“Where should we host the AI?” mixes several layers that solve different problems. Separate the interface, gateway, model serving and data layer before comparing products.

### Four layers of a deployment

| Layer | Job | Examples |

| 1. Interface | What users touch | Your application, Open WebUI, LibreChat, Teams, Slack |
| 2. Gateway | Authentication, provider routing, budgets, rate limits and logs | LiteLLM, managed gateways, cloud native gateways |
| 3. Model serving | Runs the model or exposes it as an endpoint | Provider APIs, AWS Bedrock, Microsoft Foundry, Vertex AI, serverless open weight hosts, vLLM |
| 4. Data and retrieval | Business context and evidence | Postgres, search indexes, object stores, vector databases, systems of record |

[Figure] This ends a lot of circular platform debates. A chat interface, an AI gateway, a model platform and a retrieval store solve different jobs. Compare products within the same layer before deciding whether one replaces another.

### An AI gateway becomes valuable surprisingly early

Once more than one application or team calls models, a gateway gives you one place for credentials, routing, spend controls, logs, policy and provider failover. Retrofitting that layer after every application contains its own provider key and model logic is painful.

Do not confuse a gateway with a model provider. LiteLLM can sit in front of several providers. Open WebUI can sit in front of the gateway. Bedrock or Foundry can sit behind it. Draw the layers before deciding which product overlaps which.

Current market

### The hyperscaler platforms

AWS Bedrock, Microsoft Foundry and Google Vertex AI offer model catalogues, enterprise identity, regional controls and managed AI services inside the customer's existing cloud estate. The deciding factor is often where the rest of your organisation already lives rather than one model benchmark.

Two caveats matter. Direct provider pricing can differ from cloud marketplace pricing, and the newest model may not arrive on every platform at the same time. Also verify the compliance scope for the exact AI service. A cloud provider's general certification does not automatically prove that every managed AI feature sits inside the same contractual boundary.
| Platform | Usually strongest when | What to verify |

| AWS Bedrock | Your data and operations already sit in AWS, or you want a broad managed model choice without giving applications separate provider integrations. | Exact model availability by region, feature-specific compliance scope, quotas and price versus the direct provider. |
| Microsoft Foundry | Identity, data and user workflows are already Microsoft-centred, especially where Azure AI Search, Entra and Microsoft 365 integration matter. | Which models and agent features are available in the required region, how data flows between services, and which controls belong to the AI service rather than the wider Azure estate. |
| Google Vertex AI | Your analytical estate is built around Google Cloud, BigQuery or Gemini, or multimodal and long-context workloads are central. | Regional processing, grounding architecture, model version policy, quotas and the boundary between managed Google models and third-party offerings. |

### Serverless open weight hosting

Providers such as Fireworks, Groq, DeepInfra and others expose open models through ordinary metered APIs. This is an important middle ground: open weight economics and model choice without GPU operations. For many organisations, it is the sensible first answer to “we want to use open models”.

### Rented GPUs and your own serving stack

You can rent accelerators from specialist GPU clouds or hyperscalers and run vLLM or another server yourself. That gives you control over model version, quantisation and serving behaviour while avoiding hardware ownership. You still own capacity planning, security, updates and reliability.

### Three reference architectures

Small team: sanctioned private AI quickly
A self hosted interface and gateway, protected by company SSO and HTTPS, calling a commercial provider under appropriate business terms. Logs and usage controls live centrally. If the model itself need not be local, do not buy a GPU merely to make the diagram look private.

Customer facing product
Application → gateway → one or more model providers; retrieval in Postgres or a managed search service; queue for long running jobs; traces on every request; evaluation in the deployment pipeline; secrets in the normal secrets manager. Containerise it on the platform the engineering team already knows.

Large or regulated estate
Enterprise identity, policy enforcement, per team budgets, regional or sovereign deployment constraints, permission aware retrieval, central observability, an AI inventory and documented human oversight per use case. At this scale, identity, procurement and data architecture usually consume more project time than the model API.

### What fails first at scale

- Provider quotas and rate limits. Capacity needs to be agreed before launch.

- Long lived requests. Agent runs expose load balancer, timeout and connection assumptions designed for ordinary web calls.

- Token growth. Multi step traffic can increase cost faster than user count.

- Retrieval and permissions. Large indexes, ACL filters and ingestion freshness can become the bottleneck before generation.

- Model change. Stable product names can hide updated weights or behaviour. Pin versions where possible and keep regression tests.

27 · Economics

## Token consumption and how to reduce it without reducing quality

Most AI bills come down by sending less rather than by buying cheaper, and caching a stable prefix is the single largest lever.

Token optimisation is not about making every prompt tiny. It is about spending model compute where it changes the business outcome. The metric to optimise is cost per completed task, not the cheapest price per million tokens.

### Count the hidden context

The user's visible question may be twenty tokens while the actual request contains thousands of tokens of system instructions, tools, retrieved passages, conversation history and agent state. That hidden context often dominates the bill.

Agent loops amplify the effect because later steps frequently carry state accumulated from earlier ones. Depending on how the harness compresses history and caches prefixes, total consumption can grow much faster than the visible number of user turns.
A typical retrieval-heavy request can easily include a 1,000–3,000 token standing instruction, another 1,000–3,000 tokens of tool descriptions, several retrieved passages of a few hundred tokens each, plus conversation and state. Those are planning ranges, not limits, but they explain why optimising the twenty-token user question rarely moves the bill.

[Figure] Count the request the model sees, not the message the user sees. The exact proportions vary widely, but in production RAG and agent systems the hidden context can dominate cost and latency.

### 1. Prompt caching

Many providers can reuse the internal computation for a repeated prompt prefix and bill cached input at a lower rate. This works best when static content comes first: system instruction, stable tool definitions and reusable examples before dynamic records and user specific material.

Provider prices and cache rules differ, so avoid universal savings claims. The logic is still powerful. If a large static prefix is reused repeatedly, caching can reduce both input cost and time to first token substantially. Verify actual cache hits in provider usage metadata rather than assuming the prompt is cache friendly.

Caching discounts input computation. It does not make output tokens free.
Cache arithmetic, without pretending the rate is universalSuppose a provider charges $3 per million ordinary input tokens, 1.25 times that rate to write a cache entry and 0.10 times the base rate to read it. One cache write followed by nine reads costs $3.75 + (9 × $0.30) = $6.45 for ten million-token uses of the same prefix. The effective input rate is therefore $0.645 per million, a 78.5% reduction from paying $3 ten times. The calculation is useful; the multipliers are provider-specific. Check the current cache policy before budgeting.

[Figure] The example shows the mechanism, not a universal discount. It assumes a $3 base input rate, a 1.25× cache-write multiplier and a 0.10× cache-read multiplier, exactly as stated in the accompanying text. Real providers use different rules.

### 2. Model routing

Run classification, extraction and simple transformations on the smallest model that passes your evaluation. Escalate hard cases to stronger models. The saving depends on your traffic mix and escalation rate, so measure it at task level rather than quoting a generic 60 or 80% reduction.

### 3. Batch the work that nobody is waiting for

Overnight enrichment, bulk tagging and backfills do not need interactive latency. Provider batch products and your own queues can trade time for lower cost and smoother capacity. If a workload runs on a schedule, question why it is paying interactive prices.

### 4. Engineer context deliberately

Retrieve broadly, then pass only the best evidence. Summarise old conversational turns. Strip unused fields from tool responses. Expose only tools relevant to the current step. These changes reduce cost, latency and attention competition at the same time.

### 5. Semantic caching

For repetitive support or FAQ traffic, you can embed a new query and return a previously validated answer when the meaning is sufficiently close. This saves both input and output model tokens because the model call disappears.

The risk is returning yesterday's answer to a subtly different or time sensitive question. Derive similarity thresholds from labelled examples, include freshness rules and bypass the cache for personalised or rapidly changing information.

### 6. Reasoning budget

Reasoning consumes compute and often billable tokens. Allocate it by task. A hard root cause analysis may justify substantial deliberation; extracting a serial number or formatting an email usually does not.

### The finance metric

Cost per completed task combines model price with retries, tools, failures and human corrections. A cheap model that needs three attempts can cost more than an expensive model that succeeds once. Track the AI cost beside the human time saved and the value or risk of the outcome.

Optimise the system, not the token price
Teams can often reduce spend dramatically through caching, routing, smaller context and batch processing, but the percentage depends on the starting architecture. Publish the measured saving from your own workload. Avoid turning a successful internal optimisation into a universal market claim.

28 · Data risk

## Where your data goes: consumer AI, enterprise AI and shadow use

Turning off training is one control among several, and it is not the one that decides whether the data was safe to send in the first place.

The useful governance question is not “is this model approved?” The same model can be consumed through a personal account, an enterprise contract, a private API endpoint or infrastructure you control. Classify the deployment and data path, not the brand name.

Field note

### Tier by where the data is processed

We wrote a group AI policy for a listed scientific instruments group of more than twenty-five operating companies. The framework sorts tools by where processing happens — consumer cloud, business AI under an enterprise agreement, local or self-hosted — not by which model is running. The same model can sit in different tiers depending on deployment, which is precisely what stops next quarter's release from invalidating the policy.

### A simple three tier policy

| Tier | Typical deployment | Data policy | Reason |

| 1 · Consumer | Individual consumer accounts and free services | Public and non sensitive material only | No organisation level contract, policy enforcement or central audit |
| 2 · Business / enterprise | Approved enterprise products and contracted APIs | Internal data subject to the organisation's classification and contract | Business terms, identity, retention controls, audit and processor commitments can be reviewed |
| 3 · Controlled local / sovereign | Models and services inside infrastructure the organisation controls | Potentially the most sensitive classes, subject to internal approval | Inference data can remain inside the chosen security boundary |

This is a policy framework, not a claim that every product in a tier is equivalent. A particular enterprise service may still be inappropriate because of jurisdiction, subprocessors, retention, contractual restrictions or the data itself.

Field note

### Two positions that survived board review

In that same group policy, two positions did most of the work. An opt-out setting was treated as insufficient on its own, because terms change, breaches happen and courts compel disclosure. And an enterprise agreement was held not to override a contractual obligation to a customer under NDA — the vendor's promise is made to you, not to your client.

### Turning off training is only one control

A “do not train on my data” setting answers one question: whether your content may be used to improve future models under the provider's policy. It does not, by itself, answer where the data is processed, how long it is retained, who can access it under legal process, whether your NDA permits the disclosure or whether your organisation can audit the use.

### The legal hold example, stated precisely

In 2025, litigation brought by the New York Times and other publishers led to a US court order requiring OpenAI to preserve categories of consumer and API content that would otherwise have been deleted. OpenAI states that the broad ongoing preservation obligation ended on 26 September 2025 and ordinary deletion practices resumed for new data, while a limited historical set remained preserved. Separately, in December 2025 OpenAI said it was required to make a de identified sample of 20 million consumer conversations available under strict controls in the litigation.

The durable lesson is not “every deleted chat is stored forever”. That would be wrong. The lesson is: a provider's ordinary deletion policy can be overridden by a valid legal preservation or disclosure order. Consumer privacy settings are not immunity from legal process.

### A rule staff can remember

If you would not email it to a competitor, do not paste it into an unapproved consumer AI tool
The sentence is intentionally conservative and memorable. It pushes people to stop before sharing trade secrets, NDA material, personal data, confidential pricing, unpublished financials or export controlled technical information through a service the company has not assessed.

### Categories that need controlled handling

| Information | Why consumer use is risky |

| Trade secrets, proprietary designs, unpublished R&D, source code | Confidentiality and trade secret protection depend on controlled disclosure |
| Material received under an NDA | Sending it to another processor may breach the agreement even if no human reads it |
| Personal data | Lawful basis, processor terms, transfers, retention and access controls need to be understood |
| Board material, forecasts, M&A and unpublished financials | Commercial sensitivity and, in some contexts, market abuse obligations |
| Customer contracts and negotiated pricing | Confidentiality obligations and commercially sensitive terms |
| Export controlled technical material | Where processing occurs can itself create a compliance problem |

A useful policy also states what is encouraged. Public research, drafting from non confidential material, learning, translation, ideation and coding on approved data can all be positive uses. A policy that only says “no” creates an incentive to hide use rather than improve it.

### Enterprise AI introduces permission amplification

An enterprise assistant may honour the permissions of SharePoint, Drive, CRM or another source and still expose a problem that already existed. A salary spreadsheet in an overshared folder was technically accessible yesterday; nobody knew where to look. Semantic search makes it easy to find today.

The AI did not create the access right. It removed the friction that had hidden the mistake.

- Audit oversharing before broad enterprise search is enabled.

- Apply source permissions during retrieval, not only after generation.

- Consider excluding HR, legal, corporate development and other high sensitivity areas from general indexing.

- Remember that an enterprise contract with an AI provider does not automatically override confidentiality promises you made to customers or suppliers.

### Shadow AI is an adoption problem as well as a policy problem

People use personal AI accounts for work when the sanctioned route is absent, slow or materially worse. Start by asking what is already being used without turning the discovery exercise into disciplinary action. Provide an approved tool that solves the real job. Then enforce the boundary. Prohibition without a usable alternative tends to push activity outside visibility.

### Six questions before approving a service

| Area | Ask | Warning sign |

| Residency | Where is content processed and stored, and is the commitment contractual? | A marketing claim with no matching contract term |
| Training | Are business inputs used for model training, and what is the default? | Unclear policy or no business grade opt out |
| Jurisdiction | Which entity processes the data and which government access regimes apply? | The answer conflicts with your customer or regulatory obligations |
| Contract | Is there an appropriate DPA and business agreement? | Consumer terms used for company confidential information |
| Security | What independent certifications, controls and incident process cover this service? | Claims cannot be tied to the actual product or processing scope |
| Auditability | Can admins see who used the system, which data classes and when? | No central logs, export or policy controls |

Hosted Chinese API versus Chinese open weights
These are different risk decisions. A hosted API transfers your inference data to the provider under its contractual and jurisdictional framework. Downloaded weights running inside your environment do not transmit prompts to the original laboratory simply because the weights were created there. For local weights, focus instead on licence, provenance, model behaviour and software supply chain controls.

Law & standard

### AI literacy is now a compliance issue in the EU

The EU AI Act's AI literacy obligation has applied since February 2025. The Digital Omnibus, in force since 27 July 2026, rewrote it: organisations must now take measures to support the development of AI literacy among staff and other people dealing with AI systems on their behalf, rather than guarantee a level in any individual. That is an obligation of means, not of result — the duty is softer, but it has not gone away. A practical programme includes the data boundary above, known failure modes, verification habits and records of the training delivered.

Related on this siteHow Ascentis AI handles client data, and the certifications and controls behind it: Security & confidentiality.

29 · Control

## The governance layer

Governance is what makes a higher-consequence AI system approvable, and it is mostly inventory, permissions, traces and a rehearsed way to stop.

Governance is the mechanism that lets you give an AI system more responsibility without pretending it has become predictable. Good governance expands the set of things a business can safely automate.

### Technology specific risks to put on the register

Prompt injection
Language models receive instructions and untrusted content through the same broad context channel. A malicious instruction can therefore arrive inside an email, web page, uploaded document or tool result rather than directly from the user.

Imagine an email containing hidden text that tells an assistant to forward sensitive messages elsewhere. The agent reads the email because reading email is its job. If the same system also has mailbox access and an unrestricted send tool, the attack can become an action without installing conventional malware.

Prompt injection remains a leading application security concern. Filters, delimiters and secondary models can reduce risk, but none should be treated as a complete security boundary against an adaptive attacker. Architecture matters more.

The lethal trifecta
The highest risk pattern combines private data, untrusted content and an outward communication or action channel. Remove or tightly constrain one leg and the major exfiltration path is substantially reduced. This is not a proof that every attack disappears; it is an excellent two minute design check before an agent receives new permissions.

[Figure] Architecture beats a filter-only strategy. Prompt injection cannot be treated as a solved filtering problem. This check, popularised by Simon Willison, highlights the combination that creates a particularly dangerous exfiltration path.

Excessive agency
A system can have more tools, broader permissions or more autonomy than the task requires. That turns a small model mistake or prompt injection into a large incident. Use least privilege, bounded scopes, human approval for irreversible actions and explicit action budgets.

Other recurring risks

- Confabulation: plausible but unsupported output entering decisions.

- Sensitive information disclosure: data leaking through outputs, logs, indexes or external processors.

- Data poisoning: manipulated content entering a retrieval corpus or training set.

- Supply chain: models, packages, skills, containers and MCP servers introduced without provenance or review.

- Unbounded consumption: loops, retries or malicious inputs causing runaway compute and spend.

- Permission amplification: AI making existing overshared information dramatically easier to find.

Law & standard

### EU AI Act, position as at 20 August 2026

The AI Act applies according to role and risk category, and a UK company can still fall within scope when placing or using relevant AI systems in the EU market. The implementation calendar changed in 2026, so older presentations are now easy to misread.

| Date | Key position |

| 2 Feb 2025 | Prohibited practice rules and the AI literacy obligation began applying |
| 2 Aug 2025 | General purpose AI model obligations and elements of the governance and penalties framework began applying |
| 2 Aug 2026 | Article 50 transparency duties apply and the Commission / national enforcement framework expands |
| 2 Dec 2026 | Limited transitional period ends for Article 50(2) marking obligations on systems already placed on the market before 2 Aug 2026 |
| 2 Dec 2027 | High risk rules for Annex III type systems apply under the revised timetable |
| 2 Aug 2028 | High risk rules for AI embedded in regulated products apply under the revised timetable |

[Figure] The calendar is staggered. This timeline mirrors the dated table in the article and deliberately omits a single headline penalty figure because different breaches carry different ceilings.

### Article 50 is more specific than “label all AI content”

From 2 August 2026, providers of certain interactive AI systems must design them so people are informed when they are interacting directly with AI. Providers of systems generating or manipulating synthetic content have machine readable marking obligations within Article 50's scope. Deployers have separate disclosure duties for areas including deepfakes, emotion recognition or biometric categorisation, and certain AI generated public interest text without human review or editorial control.

For Article 50 breaches, the Commission's guidance notes penalties can reach €15 million or 3% of worldwide annual turnover. The higher €35 million or 7% ceiling often quoted for the Act relates to the most serious categories such as prohibited practices, not every transparency failure.

### The UK position

The UK still does not have one horizontal AI Act equivalent. AI use is governed through existing laws and sector regulators, including data protection, equality, consumer, employment, safety and intellectual property rules. In practice, organisations trading across Europe often build one governance layer capable of satisfying both UK obligations and relevant EU requirements rather than maintaining two completely different operating models.

Law & standard

### Frameworks worth knowing

NIST AI Risk Management Framework organises work around Govern, Map, Measure and Manage. It is voluntary and not a certification. The Generative AI Profile extends it for generative AI risk.

ISO/IEC 42001 is a certifiable AI management system standard. That makes it attractive in procurement because an independent certification can provide evidence of an operating management system rather than only a self assessment.

Singapore's Model AI Governance Framework for Agentic AI, published in January 2026, is particularly useful for agent deployments because it explicitly addresses autonomy, technical controls, human accountability, staged deployment and monitoring.

Field note

### Escalate the judgement calls instead of burying them

The deliverable that mattered most on that engagement was not the policy. It was a decision pack isolating eight questions a consultant should not answer — oversight frequency, liability allocation, enforcement, the stance on HR and recruitment AI — each with a drafted position, the alternatives and the trade-off. Underneath sat an approved-tool register of around twenty tools and a banned list where every entry carried a written rationale, so the list could be defended and revisited rather than inherited.

### What a working governance layer contains

- Inventory. Every AI system, owner, purpose, model, data class, users, deployment route and risk tier.

- Policy and access controls. Approved services, allowed data classes, identity and permission boundaries.

- Runtime controls. Input checks, output checks, tool permissions, budgets and approval gates.

- Traces. Prompts or references, retrieved evidence, model version, tool calls, approvals, tokens, cost and outcome as appropriate to the risk and privacy model.

- Evaluation in the release path. Run your own tests before deployment and after material model, prompt, retrieval or tool changes.

- Human oversight matched to blast radius. Specify who reviews what and at which point.

- Incident response. Stop, contain, notify, reverse where possible, preserve evidence and feed the lesson into the evaluation set.

30 · Readiness

## Data readiness and the AI maturity curve

Generative models changed what “data ready” means: the material that matters is usually the knowledge nobody ever wrote down.

A promising AI idea often turns into a data project as soon as somebody asks where the evidence actually lives. That is not a detour. It is frequently the real work.

### Five questions before commissioning the build

- Does the data exist? Is the process recorded today, or does the knowledge live only in people's heads?

- Can you access it? Data trapped in a proprietary application, supplier portal or personal drive is functionally unavailable until you solve access.

- Is it correct enough? Sample records manually. Historical data often contains shortcuts that were harmless while nobody expected a machine to reuse them.

- Do you have the outcome labels you need? Predictive projects need the target, not merely the input. “Machine received maintenance” is not the same label as “machine failed”.

- Are you allowed to use it? Contracts, personal data, NDAs, export control and customer commitments can change the architecture or stop the use case entirely.

### Generative AI changed what “data ready” can mean

Traditional machine learning often required a structured dataset before the project could begin. Language and multimodal models can extract useful structure from messy service notes, email threads, scans, call transcripts and inconsistent reports.

That means a good first project for a poorly structured organisation may be a system that reads the mess. Searchable knowledge creates immediate value while the ingestion process starts producing the cleaner structured dataset needed for forecasting, quality models or automation later.

Field note

### Where the tacit knowledge was actually kept

On a sales adviser build, what made the system useful was twenty years of institutional knowledge that had never been treated as data: battle cards on outdated slide decks circulating in messaging groups, competitor analysis, and the reasoning behind the pricing rather than the price list. Collecting and structuring that was the project. The model was the easy part.

### Tacit knowledge is the data nobody budgeted for

In engineering SMEs, the highest value knowledge often explains why the written procedure is not quite enough: which installation needs the modified bracket, which symptom usually precedes the failure, which drawing detail changes the quotation. A few experienced people carry it.

Capture that deliberately. Structured interviews, recorded and transcribed service reviews, annotations on historical cases and “why” fields in new workflows all turn experience into an asset that retrieval and future models can use. Proprietary operational knowledge is also far more defensible than a prompt or generic model choice.

### A six level maturity model

This is an Ascentis planning framework, not an industry standard. Use it to decide what the organisation should do next.

| Level | What it looks like | Sensible next move |

| 0 · Unaware | Little use, little policy, leadership mainly sees hype or threat | Literacy and basic policy before technology procurement |
| 1 · Shadow | Individual consumer use, no central view or rules | Discover usage, provide a sanctioned route and define data tiers |
| 2 · Sanctioned | Approved tools and training, mainly individual productivity | Pick one end to end process and establish an evaluation set |
| 3 · Applied | One or more measured production systems on company data | Reuse the evaluation, retrieval and governance capabilities across further use cases |
| 4 · Systemic | AI embedded in several core processes with shared controls | Use agents and hybrid ML only where the process genuinely earns the complexity |
| 5 · Differentiating | AI capability and proprietary data contribute to the product or competitive advantage | Protect the data loop, evaluation capability and operating knowledge as core IP |

[Figure] Do not skip the operating capability between stages. A level 1 organisation can technically prototype a level 4 agent. The harder question is whether it has the data, ownership, evaluation and incident controls to keep that system alive.

### Two predictable traps

Skipping levels. A shadow AI organisation can technically build an autonomous agent, but it usually lacks the evaluation, ownership, permissions and incident process required to keep it alive in production.

Stalling at sanctioned tools. Buying licences and training people can improve individual productivity without changing a business process. Moving from level 2 to level 3 is a management decision: redesign one measurable workflow around the capability.

31 · Application

## Your first ninety days

Ninety days is enough to put one narrow, testable workflow into production, and not enough to do anything broader well.

A useful first AI programme does not begin with an agent framework. It begins with visibility, one worthwhile process and a way to measure whether the result is better than what you do today.

### Days 1 to 30: see the current state

- Ask what AI staff already use and for which tasks. Do the discovery without punishment so you learn the real picture.

- Publish a simple data tier policy and provide at least one sanctioned tool people can actually use.

- Choose one high volume process where language or documents matter and mistakes are recoverable.

- Create twenty to fifty real examples with an agreed acceptable outcome. This is the first version of the evaluation set.

- Name an owner and define which outputs require human approval.

Field note

### What sixty days actually buys

Some calibration from our own delivery. A support triage proof of concept took five weeks. A private sales adviser went from MVP to production in eight. A self-hosted weekly intelligence digest reached its first live issue in six and a half weeks against eight quoted. None were research projects. All were narrow, testable, and scoped to a single workflow.

### Days 31 to 60: build the least complicated version

- Try prompting and structured output first.

- If the system needs changing company knowledge, add retrieval or direct tools rather than training facts into the model.

- Use hybrid retrieval when exact identifiers matter; test a reranker only if the candidate set shows a ranking problem.

- Compare several model tiers against the same evaluation set and choose by task success, then cost and latency.

- Measure human correction rate. It is one of the clearest signals of whether the system is becoming operationally useful.

### Days 61 to 90: turn the prototype into an operating system

- Instrument cost per task, latency, correction rate, retrieval quality, failure categories and abstention.

- Add identity, permissions, traces, versioning and an incident path.

- Run the lethal trifecta check before enabling outward actions.

- Write down what failed and add those cases to the evaluation set.

- Only introduce an agent where the next step cannot sensibly be predetermined. Add durable state, budgets and kill switches with it.

### Where we have used the method

Our case studies cover market intelligence, group wide AI governance, a five stage support pipeline, quotation automation, market entry, a global sales adviser and product embedded AI. Client names are withheld where agreed; the architecture, process changes and measured outcomes are described so readers can see what moved the result beyond a chatbot demonstration.

Four questions to carry into every AI meeting
Which layer is failing? Prompt, context, harness or loop.

What evidence is actually in context? If nobody can answer, the system is not understood well enough.

How is success verified? “The model says it is done” is not a production control.

What is the blast radius? Autonomy should follow consequence and reversibility, not the marketing capability of the model.

Related on this siteWhat the first ninety days looked like elsewhere: Ascentis AI case studies.

32 · Questions

## Frequently asked questions

Q. Is every AI problem a job for a language model like ChatGPT?
A.No. Start with the information carrying the signal. Language and documents point towards LLMs; sensor series, tabular data, images and constrained optimisation may be better served by forecasting, classical machine learning, computer vision or operations research. Many strong industrial systems combine a specialist model for the calculation with an LLM for explanation and workflow.

Q. Why does our RAG system perform badly on scanned PDFs and datasheets?
A.Inspect the extracted content before blaming retrieval. Scans can yield no text, tables can lose row and column relationships, and diagrams can disappear. Use OCR, layout aware extraction or vision models as appropriate, preserve tables structurally, and require source location for important extracted fields.

Q. How do you make an AI return reliable structured data rather than prose?
A.Use schema constrained outputs where the provider supports them, then validate the values with deterministic business rules. Structural validity does not prove semantic correctness. A production pattern is generate, validate, repair with the specific error, then escalate after a bounded number of attempts.

Q. Our data is not ready for AI. Where do we start?
A.Ask five questions: does the data exist, can you access it, is it correct enough, do you have the labels needed for the task, and are you allowed to use it? Generative AI can help earlier than traditional ML because it can turn messy documents, emails, scans and transcripts into searchable and increasingly structured material.

Q. What is the difference between prompt engineering and context engineering?
A.Prompt engineering designs the instruction and examples. Context engineering decides which information, memory, documents, tool descriptions and intermediate results are made available to the current inference. In production systems, getting the right context is often more important than polishing one sentence of prompt wording.

Q. What is loop engineering, and did Anthropic invent the term?
A.Loop engineering is useful practitioner shorthand for designing how an agent gathers information, acts, verifies, persists state and decides whether to continue. Anthropic published influential patterns for long running agents and the gather, act, verify cycle, but the phrase itself is community terminology rather than an Anthropic standard.

Q. What is RAG in simple terms?
A.Retrieval augmented generation searches for relevant evidence when a question arrives, then gives that evidence to the language model so it can answer from it. Think open book exam rather than memory test. It is particularly useful for changing company knowledge because documents can be updated without retraining the model.

Q. Does a large context window make RAG unnecessary?
A.Sometimes a small corpus can be placed directly in context and retrieval can be skipped. At larger scale, retrieval still controls cost, latency, relevance and permissions. Large windows make it possible to supply richer evidence; they do not make information selection disappear.

Q. What is the difference between a reranker and an embedding model?
A.A conventional dense embedding retriever represents query and document separately so it can search very large collections quickly. A cross encoder reranker reads each query and candidate passage together, which often improves precision on a shortlist. Retrieve for recall first, rerank for precision second.

Q. What is Reciprocal Rank Fusion and why is k often set to 60?
A.RRF combines ranked lists using positions rather than incomparable raw scores: each result receives a contribution of 1/(k + rank) from each retriever. A k of 60 is a common convention, not a sacred default. On short lists a smaller value can create stronger separation, so tune it against your evaluation set rather than copying a blog post.

Q. What similarity threshold should I set for vector search?
A.There is no portable number. Embedding score distributions vary by model, domain and chunk length. Label relevant and irrelevant examples, inspect the distributions and choose a floor that matches the trade you want between recall and abstention.

Q. Should we self host an LLM or use an API?
A.For economics, compare the total cost of self hosting with a provider serving the same open model, not only with frontier API prices. For strategy, ask a different question: do sovereignty, offline operation, data egress rules, continuity or model control justify owning the serving path even if an API is cheaper? Self hosting is often a utilisation decision economically and a sovereignty decision strategically.

Q. What is the difference between open weight and open source AI models?
A.Open weight means the trained parameters can be downloaded under a licence. It does not guarantee that training data, training code or the full development process are open, and licences can impose conditions. Open weight also says nothing about where you run the model: local hardware, rented GPUs and serverless APIs are all possible.

Q. How do you stop an AI agent from being hijacked by prompt injection?
A.Do not depend on one filter. Minimise the combination of private data, untrusted content and outward action channels, apply least privilege to tools, sandbox code, require approval for consequential writes and monitor actions. Prompt injection is a system security problem, not only a prompt wording problem.

Q. Does turning off “use my data for training” make a consumer AI tool safe for confidential information?
A.No. That setting addresses model training, not the complete data path. Processing location, retention, contractual terms, legal process, breach exposure and your own NDAs still matter. Use approved business services for company confidential data and keep sensitive classes out of unapproved consumer accounts.

Q. What should never be entered into an unapproved public AI tool?
A.Examples include trade secrets, unpublished designs, NDA material, personal data, board and M&A material, confidential customer pricing, proprietary source code and export controlled technical information. The exact boundary should come from your organisation's data classification and contracts, not from a universal internet list.

Q. Is Microsoft 365 Copilot safe to roll out across the business?
A.Enterprise controls improve the data handling position, but broad rollout can expose permission problems you already have. Review overshared SharePoint, Teams and OneDrive content first, preserve source permissions in search, and exclude highly sensitive areas where appropriate. AI can make technically accessible information much easier to discover.

Q. Our policy bans Chinese AI services. Does that mean we cannot use Chinese open weight models?
A.Not automatically. A hosted Chinese API is a third party processing and jurisdiction decision. Downloaded weights running entirely inside your controlled environment do not send inference data to the original model laboratory merely because of the model's origin. You still need to review licence, provenance, security and model behaviour.

Q. How do you catch AI fabrication in a research summary?
A.Audit the synthesis against the source set, not only for readability. Check omissions, unsupported claims, changed figures, incorrect reconciliation where sources disagree, and drift outside scope. Treat source comparison as a separate review step and mark findings that would change a decision as critical.

Q. What retrieval numbers should I ask an AI supplier for?
A.Ask for a first stage recall or Hit@k metric, a post rerank ranking metric such as Hit@1 or nDCG, and system metrics for citation correctness, faithfulness and task success. Then ask how they were measured: question count, corpus version, gold labels, model versions, date and whether reranking was enabled.

Q. How many types of RAG are there?
A.There is no official count. Academic surveys describe broad paradigms such as naive, advanced and modular RAG, while research papers and practitioner lists name many methods. This guide uses nine practical architecture patterns because they map cleanly to different failure modes.

Q. Is RAG just a chatbot?
A.Often the user interface is a chatbot, yes. The engineering underneath is what makes the answer trustworthy: ingestion, permissions, retrieval, reranking, citations, abstention and evaluation. RAG is the information retrieval pattern, not the chat box itself.

Q. What is an AI wrapper, and are wrappers worth paying for?
A.A thin wrapper adds little more than interface and prompt around a model API. A valuable AI product can still use an external model if its value sits in proprietary data, integration, workflow, audit, permissions, evaluation or distribution. Apply the substitution test: could a competent user reproduce most of the value with a good prompt in a general assistant?

Q. How much does an AI API call cost?
A.It depends on model, input, output, reasoning, cache status and provider. In 2026 the market spans well below a dollar to tens of dollars per million tokens. The figure to manage is cost per completed task because retries, tools and human correction can outweigh the list price of the model.

Q. What is Open WebUI and do we need Bedrock as well?
A.They solve different layers. Open WebUI is an interface for users. Amazon Bedrock is a managed model platform. You might put Open WebUI in front of a gateway that calls Bedrock, a direct provider, a local vLLM server or several of them.

Q. Do we need an AI gateway?
A.If several applications or teams will call models, a gateway is usually worth introducing early. It centralises credentials, model routing, budgets, rate limits, logging and provider failover. The exact product matters less than having one policy point rather than spreading provider logic through every codebase.

Q. Why are Chinese AI models open weight and often so cheap?
A.Several forces combine: ecosystem distribution, cloud and compute economics, strategic standard setting, efficient mixture of experts architectures and intense hosting competition around downloadable weights. Low prices are both an engineering result and a market strategy, which is why they can change rapidly.

Q. What is the difference between total and active parameters in a mixture of experts model?
A.Total parameters describe the full stored model. Active parameters describe the subset routed through for a token or layer under the MoE design. Total size strongly affects memory requirements; active compute is an important driver of inference cost, although attention, memory bandwidth, routing and serving implementation matter too.

Q. Is it safe for a UK or European company to use Chinese open weight models?
A.Model nationality alone is not the data transfer. If weights run inside infrastructure you control, prompts can remain inside that environment. Assess the model and software supply chain, licence, provenance, vulnerabilities and behaviour; assess hosted APIs separately for processor terms, jurisdiction and data residency.

Q. What does the EU AI Act require as of August 2026?
A.AI literacy and prohibited practice provisions have already applied since February 2025, GPAI obligations began in August 2025, and Article 50 transparency duties apply from 2 August 2026. The revised high risk timetable applies from 2 December 2027 for Annex III type systems and 2 August 2028 for high risk AI embedded in regulated products. Article 50 duties differ for providers and deployers, so “label all synthetic content” is too crude a summary.

Q. How much can we realistically cut our AI running costs?
A.There is no universal percentage. Prompt caching, model routing, smaller context, batching, semantic caching and reduced reasoning can each produce large savings on the right workload. Measure the before and after cost per completed task and publish your own result rather than applying somebody else's headline reduction.

Q. When should an agent drive a computer screen rather than call an API?
A.
When the application has no usable API, or when building and maintaining the integration would cost more than a supervised GUI agent. Treat the screen as a fragile, untrusted interface: an isolated session, constrained destinations and credentials, a state check after every action, idempotent writes, and human approval before anything irreversible.

Q. What is the difference between a realtime voice model and a speech-to-text chain?
A.
A cascaded architecture runs speech-to-text, then a text model with tools, then text-to-speech. It is modular and easy to inspect, but it adds latency. Native realtime voice keeps the audio inside one low-latency multimodal session, which improves interruption and flow but demands more discipline about transcripts, traces and controls.

Q. What does “model supply chain” mean in a sovereign architecture?
A.
The weights are only one dependency. Tokenizer, configuration, container, inference server, drivers, packages, secondary models, tools, registries, telemetry and update channels all shape the system. A server you own is not sovereign if it still pulls unpinned artefacts or silent updates from the internet. Pin, verify, mirror, test, and promote changes explicitly.

Q. How should deletion work in a RAG system?
A.
It has to propagate from the source to every representation that could resurface the information: parsed copies, chunks, vectors, lexical indexes, graph nodes, caches and derivatives. Keep a stable source identifier, block retrieval immediately with a tombstone, purge asynchronously with retries, then verify that no live path returns the data. Backups and logs under a retention obligation follow their own policy.

33 · Reference

## Glossary

- A2A: Agent to Agent protocol, an interoperability approach for agents discovering and communicating with other agents.

- Agent: A system in which a model can choose actions, use tools, inspect results and decide what to do next in a loop.

- Agentic RAG: Retrieval exposed as a tool inside an agent loop, allowing repeated searches and source selection.

- ANN: Approximate nearest neighbour search, trading a measurable amount of recall for much faster vector lookup.

- Attention: The transformer mechanism that lets the model weight relationships between tokens or representations when computing the next step.

- Bi encoder: A retrieval design that represents query and document separately so documents can be indexed in advance.

- Blast radius: The worst credible consequence if an automated action is wrong, combined with how detectable and reversible it is.

- Chunk: A retrievable unit cut from a larger source document. Chunk boundaries set the smallest evidence unit RAG can return.

- Computer use agent: An agent that operates a graphical interface through screenshots, mouse and keyboard rather than only through APIs.

- Context engineering: Designing which instructions, evidence, memory, tools and state are available to each model inference.

- Context window: The maximum token capacity available to a model for the current inference.

- Continuous batching: Serving requests together while replacing completed sequences with new work so GPU capacity stays occupied.

- Corrective RAG: A retrieval pattern that grades evidence before generation and takes corrective action when the evidence is inadequate.

- Cosine similarity: The cosine of the angle between two vectors. Its practical score distribution depends on the embedding model and corpus.

- Cross encoder: A reranking model that reads query and candidate together to judge their relevance more precisely.

- Data residency: The physical or contractual location where data is processed and stored.

- Data sovereignty: The degree to which an organisation can operate and govern a critical data or AI capability under controls it owns.

- Deletion propagation:
Removing source information from every derived representation of it — chunks, vectors, indexes, graphs and caches — with backups and retention obligations handled separately.

- Distillation: Training a smaller model to reproduce useful behaviour from a stronger teacher model or dataset.

- Document AI: Technology for extracting structured meaning from documents, including OCR, layout analysis and vision language models.

- Embedding: A numerical representation used to compare semantic or other learned similarity between inputs.

- Few shot prompting: Including representative input and output examples in the context to demonstrate the desired behaviour.

- Fine tuning: Additional training that changes model behaviour using task or domain examples. It is usually better for behaviour than for changing factual knowledge.

- Gradient boosting: A family of supervised machine learning methods that remains very strong on many structured tabular prediction problems.

- Groundedness: The extent to which an answer's claims are supported by the evidence supplied to the model.

- Guardrail: An automated check or policy around model inputs, outputs or actions. Individual guardrails should be assumed fallible.

- Hallucination: Fluent model output that presents unsupported or false content as if it were reliable.

- Harness: The runtime around a model: tools, permissions, sandbox, credentials, validation, budgets and logging.

- Hit@k: Whether at least one gold relevant item appears within the first k search results.

- HNSW: Hierarchical Navigable Small World, a graph based approximate nearest neighbour index widely used for vectors.

- Hybrid retrieval: Combining semantic vector search with lexical or exact matching and fusing the result lists.

- Inference: Running a trained model to produce predictions, text, tool calls or other outputs.

- JSON mode: A generation mode that guarantees valid JSON syntax but not necessarily the correct schema or values.

- Jurisdiction: The legal regimes and government access powers that can apply to a provider or processing operation.

- Knowledge graph: Stored entities and relationship instances that can be traversed and queried explicitly.

- KV cache: Inference memory that stores attention keys and values for active sequences so the prefix does not need to be recomputed at every token.

- Lethal trifecta: The risky combination of private data, untrusted content and an outward action or communication channel.

- LiteLLM: An AI gateway that can present one common API in front of multiple model providers and deployments.

- Localisation: A rule requiring data or processing to remain within a defined territory or environment.

- Matryoshka embeddings: Embeddings trained so shorter prefixes of the vector retain useful information, allowing dimension truncation to be tested.

- MCP: Model Context Protocol, a common interface for exposing tools and resources to compatible AI clients.

- Mixture of experts: A model architecture that routes tokens through only a subset of specialised network components rather than activating the full parameter set every time.

- Model supply chain:
The full dependency chain behind an AI runtime: weights, tokenizer, configuration, containers, inference server, drivers, packages, secondary models, tools, registries, telemetry and updates.

- MRR: Mean reciprocal rank, averaging 1 divided by the position of the first relevant result across queries.

- MTEB: Massive Text Embedding Benchmark, a broad suite and leaderboard for evaluating embedding models across tasks.

- Multimodal: Handling more than one modality, for example text plus image, audio or video.

- nDCG@k: A ranking metric that rewards highly relevant results appearing near the top while supporting graded relevance.

- Ontology: A formal model of entity types, relationships, constraints and their meanings.

- Open weight: A model whose trained parameters are downloadable under a licence. This is not automatically equivalent to open source.

- Orchestration: The plumbing coordinating models, tools, state, retries, routing, traces, budgets and caching.

- Parameters: The learned numerical weights inside a model. Parameter count influences memory and compute but does not alone determine capability.

- Permission amplification: AI making existing overshared information dramatically easier to discover and combine.

- pgvector: A Postgres extension adding vector data types and similarity indexes to an existing relational database.

- Precision@k: The proportion of the first k retrieved results that are relevant.

- Prompt caching: Reusing computation for a repeated prompt prefix so repeated input can be cheaper and faster.

- Prompt injection: Instructions embedded in untrusted content that attempt to influence a model or agent outside the user's intended task.

- Quantisation: Representing model weights or vectors at lower numerical precision to reduce memory and sometimes accelerate computation.

- RAG: Retrieval augmented generation, supplying relevant external evidence to a generative model at query time.

- Realtime voice agent:
A voice system that listens, reasons, calls tools and answers at low latency while handling turn-taking and interruption; it may be cascaded or natively speech-to-speech.

- Recall@k: The fraction of all relevant items found within the first k retrieved results.

- Reranker: A second stage ranking model that rescored retrieved candidates against the query before evidence reaches generation.

- RRF: Reciprocal Rank Fusion, combining ranked lists by position rather than trying to add incompatible raw scores.

- Sandbox: An isolated execution environment with restricted filesystem, credentials and network access.

- Schema constrained output: Generation restricted during decoding so the result must match a declared structure.

- Self RAG: The specific research approach that trains retrieval and self reflection behaviour into the model.

- Semantic caching: Reusing a previously validated answer when a new query is sufficiently similar and freshness rules allow it.

- Shadow AI: Work use of unapproved AI services or accounts outside organisational logging, contracts and controls.

- Similarity threshold: A minimum relevance or maximum distance rule used to reject weak retrieval results; it should be derived from labelled data.

- System prompt: Standing instructions supplied by the application to shape model behaviour.

- Tacit knowledge: Experience and judgement held by people but not yet captured in a reusable organisational record.

- Taxonomy: A hierarchy of categories or concepts, simpler than a full ontology.

- Temperature: A sampling parameter that changes how concentrated the next token probability distribution is.

- Token: A basic unit processed by a model, which may be a word, part of a word, punctuation or another encoded element.

- Top p: Nucleus sampling, restricting generation to the smallest candidate set whose cumulative probability reaches p.

- Vector database: A database or search engine capable of storing vectors, building similarity indexes and applying filters under query load.

- Vision language model: A multimodal model that can interpret images or document pages together with language instructions.

- vLLM: A high throughput inference server for language models, designed around efficient scheduling and KV cache management.

- VRAM: Accelerator memory used to hold model weights, KV cache and inference working state.

- Wrapper: An application whose AI generation is provided by an external model API; whether it is “thin” depends on the value in data, workflow and integration around the call.

- Zero data retention: A provider arrangement in which eligible API inputs and outputs are not retained beyond the processing required to serve the request, subject to the exact service terms.

34 · Sources

## Where to read further

This guide mixes standards, primary vendor documentation, academic work and clearly labelled Ascentis production judgement. For anything time sensitive, use the dated source rather than treating this page as a permanent price list.

### Models, context and agents

- Building effective agents — Anthropic. A useful workflow versus agent framing.

- Effective context engineering for AI agents — Anthropic Engineering. Context selection, attention budgets and progressive disclosure.

- Effective harnesses for long running agents — Anthropic Engineering, 2025. Durable handoffs and long horizon execution patterns.

- Long running agents — Addy Osmani. Independent synthesis of long horizon agent engineering.

### Retrieval and RAG

- Retrieval augmented generation for knowledge intensive NLP tasks — Lewis et al. The paper that introduced the RAG formulation.

- Retrieval augmented generation for large language models: a survey — Gao et al. Source for the naive, advanced and modular framing.

- Self RAG — Asai et al. The specific self reflection method behind the name.

- Corrective retrieval augmented generation — Yan et al.

- Adaptive RAG — Jeong et al.

- Contextual retrieval — Anthropic. Measured contextual chunking and reranking approach.

- MTEB leaderboard — Hugging Face. Embedding shortlisting, not a substitute for domain evaluation.

- pgvector documentation — official project. HNSW configuration and current defaults.

- Hybrid search ranking with RRF — Microsoft Learn. RRF mechanics in a production search platform.

- RAGAS documentation. Open evaluation approaches for retrieval and grounded generation.

### Protocols, infrastructure and self hosting

- Model Context Protocol — specification and documentation.

- A2A protocol — agent interoperability specification.

- A First Measurement Study on Authentication Security in Real World Remote MCP Servers — Zhou et al., May 2026. 7,973 live servers found; 40.55% exposed tools without authentication.

- vLLM documentation — production inference serving.

- Ollama documentation — local model execution, parallel requests and memory behaviour.

- Hugging Face model hub — model cards, weights and licences.

### Open weight market and model economics

- Kimi K3: Open Frontier Intelligence — Moonshot AI, July 2026. 2.8T total, 104B active, native vision and 1M context.

- DeepSeek API pricing — official pricing documentation. Check live rates before budgeting; pricing changed in August 2026.

- Anthropic statement on Fable 5 and Mythos 5 access — 12 June 2026.

- Anthropic, Redeploying Fable 5 — 30 June 2026. Primary source for the lifting of the June export controls.

- Artificial Analysis — independent model price, speed and capability comparisons. Useful for rechecking time sensitive market tables.

### Security and governance

- OWASP GenAI security guidance — threat taxonomy and mitigations for LLM applications.

- The lethal trifecta — Simon Willison. The private data, untrusted content and external communication design check.

- MITRE ATLAS — adversarial threat knowledge base for AI systems.

- NIST AI Risk Management Framework — Govern, Map, Measure and Manage.

- ISO/IEC 42001 — AI management system standard.

- Singapore Model AI Governance Framework for Agentic AI — IMDA, January 2026.

### EU and UK regulation

- European Commission AI Act implementation overview — current timeline and high risk dates.

- Article 50 transparency guidelines — European Commission, July 2026.

- Article 50 questions and answers — European Commission. Includes the limited transition to 2 December 2026 and transparency penalty band.

- Regulation (EU) 2024/1689 — the AI Act text.

- UK AI regulation briefing — House of Commons Library. Existing law and regulator led UK approach.

- ICO guidance on AI and data protection.

### Privacy and legal process

- OpenAI response to New York Times data demands — updated October 2025. Records the end of the broad preservation obligation on 26 September 2025 and standard retention position.

- OpenAI litigation update — includes the December 2025 position on the 20 million de identified conversation sample. Primary source from one party to ongoing litigation; read accordingly.

### Benchmarks and factuality

- Vectara next generation hallucination leaderboard — November 2025. A harder grounded summarisation benchmark with more than 7,700 articles.

- Stanford AI Index 2026 — broad state of the field evidence and responsible AI reporting.

- SWE bench, Terminal Bench, and ARC Prize — benchmark methodology and leaderboards.

### Beyond language models

- Forecasting: Principles and Practice — Hyndman and Athanasopoulos. A standard open reference for forecasting.

- scikit learn user guide — practical reference for classical supervised and unsupervised machine learning.

- JSON Schema — specification underpinning many structured output contracts.

Keep the date in view. This article reflects the field as checked in August 2026. The engineering principles should age far more slowly than model names, prices and benchmark positions. Recheck primary sources before using a numerical figure in a decision, budget or proposal.

### About the author

Erwan Lhermitte founded Ascentis AI with CTO Meryem Kesmi to build and advise on applied AI for engineering and manufacturing SMEs in the UK and France. He trained as an electronics and computer science engineer, later completing an MBA with Distinction at Warwick Business School. Before Ascentis, he spent 23 years in industry, including ten years leading UK engineering businesses at Spherea UK, then part of the Airbus group, and within the Judges Scientific portfolio. His advisory work with industrial leadership teams spans more than eight years across Airbus Defence & Space, Spherea and Ascentis. He also serves, on a voluntary and unpaid basis, as a French government appointed Ambassadeur IA for industry and agriculture in the Direction générale des entreprises Osez l’IA network. Ascentis AI holds Cyber Essentials certification and aligns relevant practices with JSP 936, DEFCON 658 and ISO/IEC 27001 controls.

Contact: ascentis-ai.com/about · LinkedIn

### Cite this page

Lhermitte, E. (2026). Understanding AI in 2026: from prompts and RAG to agents and sovereign AI. Ascentis AI, 20 August 2026. https://ascentis-ai.com/understanding-ai-2026/

Ascentis AI · Applied AI Foundations · Corrections and challenges to anything on this page are welcome and will be credited.