Retrieval-Augmented Generation: grounded answers, source citations, audit trail.

Retrieval-Augmented Generation is not a model. It is a six-stage information-retrieval pipeline that sits in front of a language model and decides which passages of text the model is allowed to read before it writes anything. The corpus is chunked into passages, each passage is embedded into a high-dimensional vector by a separate encoder model, and those vectors are indexed in an approximate nearest-neighbour structure (typically HNSW). At query time the question is embedded into the same vector space, the index returns the top-k most similar passages, a cross-encoder reranks them, and only the surviving passages are injected into the prompt. The language model never sees the corpus. It only sees what the retrieval stack chose to show it.

Two design decisions dominate quality. The first is chunking - fixed-size, recursive, parent-document or propositional - which determines whether a passage carries enough context to answer the question on its own. The second is hybrid retrieval, where a dense semantic search is fused with a sparse BM25 token match. Dense retrieval finds conceptual matches; BM25 catches exact tokens that dense embeddings smooth over, which in the buy-side context means ISINs, LEIs, regulator codes, fund codes and dates. Production-grade RAG in regulated finance is almost always hybrid, with a reranker on top.

The 'augmented' in RAG refers to the prompt, not the model weights. Nothing is being learned at inference time. What changes is the evidence the model is conditioned on, and which retrieved chunk identifiers can be logged next to the answer.

The expensive question at a VV or boutique asset manager is rarely 'what is the answer'. It is 'where in the documents did the answer come from'. Analysts open prospectus_ABC_2024.pdf, ctrl-F for a clause, paste the paragraph into an IC memo, and re-type the page number into a footnote. KIID extraction, MiFID suitability checks, SFDR Article 8 vs 9 disclosure work, and ad-hoc client questions on a holding all run on this pattern. The cost is not the reading - it is the certainty and the citation.

RAG replaces the manual ctrl-F with a vector query, and replaces the hand-typed footnote with a structured retrieval record. The buyer-visible difference is that every sentence in a generated answer can be expanded to the exact passage it came from, with document name and page number attached, recorded as a row in the audit log at the moment the answer was produced.

The use cases where this earns its keep are the repetitive ones: regulatory-disclosure lookups, fund-policy questions from sales, internal what-does-our-process-say-about queries, and any drafting task where a junior analyst was previously the lookup engine. The senior analyst still reviews. The lookup work moves.

DORA Art. 28 requires firms to maintain a register of ICT third-party arrangements that materially support critical or important functions. The EU AI Act Annex IV section 3 requires the Technical File for a high-risk AI system to document the data used to train, validate and operate the system, including provenance. A RAG system makes both of these obligations tractable because the cited chunk identifier is the provenance record - every generated answer points back to a specific passage in a specific document version, and that pointer is the audit row.

Pair this with DORA-conformance and a data-in-house deployment: the encoder, the index, the reranker and the language model all run inside the firm perimeter or a sovereign tenancy you control. Prompts, retrieved chunks and completions do not leave the perimeter. The regulator's read of an AI-assisted answer is then the same as the read of an analyst-written one: the source can be opened, the version can be reconstructed, the chain of custody is intact.

We start with the corpus, not the model. Prospectuses, KIIDs, factsheets, internal procedure manuals, regulator circulars and historic IC memos are normalised, segmented and stamped with version and effective-date metadata. Chunking is recursive with parent-document overlap, so a retrieved passage carries its surrounding context. We embed with an encoder we can run locally, store vectors in a perimeter-resident index, and pair every dense lookup with a BM25 channel so that ISINs and LEIs survive the embedding step.

Reranking is a cross-encoder pass over the top 50-200 candidates, scoring full query-document attention rather than the pooled similarity used at retrieval time. This is the step that turns 'roughly relevant' into 'this is the passage the analyst would have quoted'. The reranked top-3 to top-6 chunks are then assembled into a structured prompt with the language model behind a host that enforces tool consent and writes a structured audit row for each call.

The language model itself is selected for what it has to do, not for marketing reach. For most drafting tasks an open-weight model in the 8B-70B range, served locally, is sufficient and keeps the entire pipeline inside the perimeter - which is the part that matters for DORA Art. 28-30 and BaFin's December 2025 AI-as-ICT-risk guidance. Where a frontier model is needed, the call is gated, logged, and constrained to non-sensitive payloads.

Every generated answer ships with chunk identifiers, model version, retrieval scores and timestamp. The citation is not a footnote - it is the audit trail by construction.

Do not buy 'the LLM searches your documents'. The language model does not search anything. A separate retrieval system selects the passages and the model only reads what the retriever returned. If a vendor cannot explain that distinction in the first ten minutes, the rest of the deck will not survive contact with your audit team.

Do not skip the reranker. Cosine similarity on pooled embeddings is a recall step, not a precision step, and in finance the wrong-but-similar passage is more dangerous than the missing passage.

Do not treat chunking as an afterthought. Fixed-size 512-token chunks cut sentences and destroy table context; the retrieval looks fine in demos and fails on real prospectuses.

Do not let the index live outside the perimeter. A managed vector database in a foreign jurisdiction is an ICT third-party concentration risk under DORA, regardless of how convenient the API is. Keep weights, vectors and prompts in a perimeter you can point to.