Local / Private LLMs

Contracts, patient records and unpublished specifications often mustn't go to a public model service. A language model that runs on infrastructure you control keeps prompts and document content inside your own network, and you decide which model and version answers. The TX Text Control AI preview runs such models with llama.cpp, on a developer workstation, on a GPU server of your organization or in a separate AI service.

The six packages of the preview and how they work together are introduced in Text Control AI. The TX Text Control AI packages are a preview, version 0.1.0-beta.3 at the time of writing: their APIs and behavior may change before the stable release, and they require appropriately licensed TX Text Control products.

What Local Means

"Local" doesn't have to mean the computer the user works on. It means that the owner of the application controls where inference takes place: on a workstation, on a private server or in an approved environment of the organization. That control has practical benefits:

• Data processing. Prompts, retrieved passages and document content can stay within your approved network boundary.
• Versions. You decide when to change the model, its quantization, the runtime or the configuration, which keeps tests repeatable.
• Availability. Once dependencies, models and runtimes are provisioned, inference doesn't depend on a public model API.
• Costs. You use your own compute capacity instead of paying a provider per request. Hardware, administration, energy and maintenance still cost money.
• Scale. A developer starts with a compact model, and a shared service uses more powerful hardware.

A local model doesn't make an application secure, compliant or offline by itself; Data Protection and Review lists what remains your responsibility.

How a Local Model Runs

A local model is a chain of four parts:

1. A GGUF file holds the weights and metadata of the model.
2. llama.cpp is the native inference engine. It runs the model in a separate llama-server process.
3. TXTextControl.AI loads the model, starts and owns the process and creates chat sessions.
4. IChatClient of Microsoft.Extensions.AI is the standard .NET interface through which your application talks to the model.

TXTextControl.AI is the package your application references; TXTextControl.AI.LlamaServer, which manages the native process, the runtime installation and hardware reporting, comes with it. Neither package contains model weights, the native llama-server executable or GPU drivers, and restoring the packages or starting the application doesn't download a model.

Provide the Runtime

The native engine is provisioned separately from the packages. There are three ways:

• Install it once. LocalRuntime.InstallAsync downloads the pinned llama.cpp release, verifies its size and SHA-256 hash, extracts it and tests it. This is a setup step, not something to do at every application start.
• Use an existing executable. Set LocalModelOptions.LlamaServerExecutablePath to its absolute path, or the environment variable TXTEXTCONTROL_AI_LLAMA_SERVER.
• Allow an automatic download. With RuntimeAcquisition = RuntimeAcquisitionPolicy.DownloadIfMissing, loading a model installs a missing runtime. The default, NeverDownload, never accesses the network for a runtime.

The managed installation uses a fixed llama.cpp release, b10621, downloaded from the llama.cpp releases on GitHub:

On Linux, the runtime needs libgomp.so.1, from the libgomp1 package on Ubuntu, even for CPU inference. Installed runtimes are cached under the local application data folder of the running account, for example %LOCALAPPDATA%\TXTextControl.AI\runtimes\llama.cpp on Windows; for services and containers, set RuntimeCacheDirectory to a persistent, writable folder. To download from an internal mirror instead of GitHub, set RuntimeDownload to a RuntimeDownloadOptions object with the mirror's BaseUrl; the mirror must serve identical files. Once the runtime and the model are in place, inference runs offline.

Load a Model

Models are GGUF files that you obtain separately, in compliance with their licenses. The blog article on running a local LLM uses Qwen_Qwen3.5-4B-Q4_K_M.gguf, a quantized GGUF conversion of Qwen3.5-4B downloaded from Hugging Face, of about 3 GB. The following console program installs the runtime, reports what it found, loads the model and asks a question:

using Microsoft.Extensions.AI;
using TXTextControl.AI;

LocalModelOptions options = new LocalModelOptions
{
    HardwareBackend = HardwareBackend.Cpu,
    ContextSize = 8192,
    MaxOutputTokens = 1024
};

// Setup step: check the machine without network access, then install the runtime
var status = await LocalRuntime.InspectAsync(options);
Console.WriteLine($"{status.Platform}, llama.cpp {status.Version}, recommended backend: {status.RecommendedBackend}");

if (status.InstallationBlocker is not null)
{
    Console.WriteLine(status.InstallationBlocker);
    return;
}

await LocalRuntime.InstallAsync(options);

// Load the model once and reuse it
await using var model = await LocalLanguageModel.LoadAsync(
    Path.GetFullPath("Models/Qwen_Qwen3.5-4B-Q4_K_M.gguf"), options);

Console.WriteLine($"Model: {model.Info.Name}, runtime: {model.Runtime.Engine} / {model.Runtime.Backend}");

using var session = model.CreateSession();
var response = await session.GetResponseAsync("Explain a document approval workflow.");
Console.WriteLine(response.Text);

The code works as follows:

• LocalRuntime.InspectAsync reports the platform, the pinned version and a recommended hardware backend without downloading anything. InstallationBlocker names a missing requirement, such as libgomp.so.1 on Linux.
• LocalRuntime.InstallAsync installs the runtime. Later runs reuse it, so a production application loads with the default NeverDownload and skips this step.
• LocalLanguageModel.LoadAsync starts the llama-server process with the model. Relative paths are resolved against the application folder and then the working directory, so an absolute path avoids surprises. Disposing the model stops the process.
• model.Info describes the model, and model.Runtime the engine and the backend in use.
• CreateSession returns a chat session, an IChatClient. Load a model once and create sessions from it, instead of loading it for every request.

The most important LocalModelOptions and their defaults:

Choose Hardware and Model Size

• The file size isn't the memory requirement. The context cache, working buffers, GPU offloading, other applications and concurrent requests need memory, too.
• Start small. For development with small models, a modern multicore CPU, an SSD and 16 to 32 GB of RAM are a practical starting range, with the upper end when the Document Editor, the MCP document server, a development environment and embeddings run together. These are planning suggestions, not certified requirements.
• CPU inference works, but measure the latency for your task. Use a backend and driver combination that you have validated with your GPU and your model; on Windows x64 with a compatible NVIDIA GPU and driver, set HardwareBackend to Cuda.
• Increase the context step by step. Start with a context of 8,192 tokens and increase it after measuring memory use and how much of your documents a request covers. Don't raise it beyond the memory or the context the model supports.

Tool calling, structured output and complex instructions depend on the capabilities of the model and runtime. A successful load says nothing about the quality of the answers, so test a model with your own documents and tasks before you choose it.

Where Inference Runs

• In your application. LocalLanguageModel.LoadAsync starts and owns the inference process on the same machine, as in the program above.
• On an existing llama-server. LocalModelOptions.LlamaServerEndpoint connects to a llama-server that runs elsewhere, for example on a GPU server, with LlamaServerModelId and LlamaServerApiKey. Protect that endpoint with TLS and authentication, and don't expose the raw inference port.
• In a separate AI service. The AiService sample hosts inference, runtime management, embeddings and knowledge collections independently of the website, which forwards requests over an authenticated connection. Several departments can share one controlled model deployment this way; it's not an automatic multi-GPU cluster, so run a single service instance.

The samples repository txtextcontrol-ai-samples describes the remote service, runtime administration and download mirrors in its docs folder.

Hosted Models as an Alternative

Since version 0.1.0-beta.2, TXTextControl.AI can also connect to OpenAI and OpenAI-compatible APIs with OpenAIChatClientFactory. Local inference remains the default, and there is no automatic fallback to a cloud service. A hosted model receives the prompts and the document text included in them, so the benefits of local processing described above don't apply to it. Older blog articles, such as PDF Document Classification with OpenAI and TX Text Control in C#, send document text to OpenAI's hosted API.

The blog article Running a Local LLM in .NET C# with TX Text Control AI builds an interactive console chat with a local model step by step.

Next Steps

• Chat with Documents: ask a local model questions about a document.
• Text Control AI: the packages and samples of the preview.
• RAG: answer questions from a collection of approved documents.