Local inference: when 'where your data goes' is the entire story.

Local inference means running an open-weight language model - Llama, Mistral, Qwen, DeepSeek, Phi, Gemma and their derivatives - on hardware inside the firm's perimeter. The perimeter is concrete: the firm's data centre, a sovereign-cloud tenancy under a contract the firm controls, or a GPU rack on-prem. No prompt, no retrieved chunk, no completion and no embedding crosses the boundary outward. There is no API call to a foreign hyperscaler in the request path.

Two layers determine viability. The first is quantization, the practice of reducing weight precision from native fp16 or bf16 to smaller formats so a model fits in available GPU memory. The dominant formats today are GGUF (a container around k-quant recipes such as Q4_K_M and Q5_K_M, used by llama.cpp, supporting CPU plus GPU hybrid execution), AWQ (activation-aware INT4 quantization that preserves the small set of weights carrying most of the output signal), and native INT8 or FP8 on H100-class hardware. A Q5_K_M quantization of a 70B model fits in roughly 50 GB of VRAM with usable context, and loses only a few percentage points of quality versus the unquantized baseline.

The second layer is the inference server. The production choice is vLLM, whose PagedAttention mechanism partitions the attention key-value cache into fixed-size blocks addressed through a per-sequence block table - effectively virtual memory for attention. This collapses KV-cache fragmentation from 60-80% waste to under 4% and lifts throughput by an order of magnitude versus naive serving on the same GPU. llama.cpp is the alternative for single-machine or CPU-plus-GPU hybrid setups; TGI is Hugging Face's comparable production server.

The compliance question that stops most AI projects at a VV or boutique manager is not 'is the model accurate'. It is 'where does the data go when we use it'. Prospectuses contain mandate detail. KIIDs contain product structure. IC memos and audit responses contain the firm's thinking. Sending any of these to a foreign frontier-model API places that content under another jurisdiction's reach, raises ICT third-party concentration risk under DORA, and - in the BaFin reading - introduces AI as an ICT risk that has to be managed accordingly.

Local inference removes the question. The weights sit on disk the firm controls. The GPU is in a rack the firm can physically point to. Prompts and completions stay inside the network boundary. Retrieval, agent loops and MCP servers run alongside the model in the same perimeter. The buy-side outcome is that an AI capability - drafting, extraction, retrieval-grounded answers, structured parsing - can be deployed against the firm's most sensitive material without changing the firm's data residency posture.

Use cases where this is decisive: prospectus and KIID extraction, IC-memo drafting, internal compliance Q and A over the firm's policies, regulatory-filing preparation, and any workflow where the cost of a single content leak exceeds the cost of operating the GPUs for a year.

DORA Art. 28-30 requires firms to manage ICT third-party risk, including concentration. The simplest answer is to avoid the third party in the request path entirely. BaFin's December 2025 guidance treats AI as an ICT risk to be managed under the existing DORA framework, with explicit attention to data flows. CSSF and FINMA expectations on data residency for Luxembourg- and Swiss-domiciled vehicles point in the same direction. The EU AI Act adds technical-file obligations on data provenance and lifecycle.

A local-inference deployment converts these obligations into a visible property of the architecture. The regulator can trace the data flow: in stays in, out does not happen. The vendor relationship reduces to the hardware supplier and the open-weight model release - both substitutable, neither in the request path at runtime. The AI capability is paired with DORA-conformance and a data-in-house guarantee not as a marketing claim but as a property of where the GPUs are racked.

We start with a hardware sizing exercise against the actual workload. A 7B-13B model on a single L40S or 4090 handles most drafting and extraction tasks at production latency. A quantised 70B-class model on a single H100 80GB serves heavier retrieval and reasoning workloads with 16-32k token context at usable batch sizes. We size for steady-state batch and peak concurrency, not for a single user, because the economics flip at batch.

Models are served with vLLM behind an internal gateway. Quantization format is selected per workload - Q5_K_M or AWQ-INT4 for the 70B class on a single H100, native bf16 or FP8 where the workload justifies it. Weights live in a versioned registry inside the perimeter; a model swap is a deployment, not a vendor change. We pin model and tokenizer versions per release and write the versions into every audit row, so the EU AI Act Annex IV section 6 lifecycle log is generated by the runtime rather than maintained by hand.

Around the inference server we wire the rest of the perimeter-resident stack: the retrieval index, the MCP servers, the agent runtime with its approval gates, the structured audit log. The boundary is one network zone; the language model is one process inside it. No outbound call to a public model provider exists in the request path.

For workloads where a frontier model is genuinely required - rare in regulated drafting work - we operate a separate gated egress with payload constraints and an explicit deviation log. The default is local. The deviation is the exception that must be argued.

Do not accept 'private AI' marketing that still routes through a managed gateway. A managed AI service hosted by a foreign hyperscaler, regardless of region, is not local inference. The weights are not yours, the request path is not yours, and the ICT third-party concentration risk under DORA Art. 28 is unchanged.

Do not confuse edge, local and on-prem. Edge means inference on a client device. Local means inside your network. On-prem means inside hardware you own. They have different threat models and different cost curves. Pick the one your regulator letter actually describes.

Do not skip the quantization choice. A naive bf16 deployment of a 70B model on hardware sized for Q5_K_M will simply fail under load, and the team will conclude that local inference 'does not work'. The quantization recipe is part of the design, not an afterthought.

Do not let the GPU rack become an undocumented dependency. It belongs in the DR plan, the BCM register and the ICT inventory like any other critical infrastructure. The point of local inference is to bring the model inside the existing control framework, not to create a parallel one.