# Local / Private LLMs

> Run language models on hardware you control with the TX Text Control AI preview: llama.cpp runtime, GGUF models, LocalModelOptions, hardware and AI services.

- **Product:** TX Text Control .NET Server
- **Section:** Documentation
- **Kind:** Topic
- **Path:** TX Text Control .NET Server > AI Integration > Local / Private LLMs
- **2794 words**
- **Web URL:** https://docs.textcontrol.com/tx-text-control-dotnet-server/ai/local-llms/
- **LLMs URL:** https://docs.textcontrol.com/tx-text-control-dotnet-server/ai/local-llms/llms.txt
- **LLMs-Full URL:** https://docs.textcontrol.com/tx-text-control-dotnet-server/ai/local-llms/llms-full.txt

---

Contracts, patient records and unpublished specifications often mustn't go to a public model service. A language model that runs on infrastructure you control keeps prompts and document content inside your own network, and you decide which model and version answers. The TX Text Control AI preview runs such models with llama.cpp, on a developer workstation, on a GPU server of your organization or in a separate AI service.

The six packages of the preview and how they work together are introduced in [Text Control AI](https://docs.textcontrol.com/tx-text-control-dotnet-server/ai/text-control-ai/llms-full.txt). The TX Text Control AI packages are a preview, version 0.1.0-beta.3 at the time of writing: their APIs and behavior may change before the stable release, and they require appropriately licensed TX Text Control products.

### What Local Means

"Local" doesn't have to mean the computer the user works on. It means that the owner of the application controls where inference takes place: on a workstation, on a private server or in an approved environment of the organization. That control has practical benefits:

- **Data processing.** Prompts, retrieved passages and document content can stay within your approved network boundary.
- **Versions.** You decide when to change the model, its quantization, the runtime or the configuration, which keeps tests repeatable.
- **Availability.** Once dependencies, models and runtimes are provisioned, inference doesn't depend on a public model API.
- **Costs.** You use your own compute capacity instead of paying a provider per request. Hardware, administration, energy and maintenance still cost money.
- **Scale.** A developer starts with a compact model, and a shared service uses more powerful hardware.

A local model doesn't make an application secure, compliant or offline by itself; [Data Protection and Review](https://docs.textcontrol.com/tx-text-control-dotnet-server/ai/overview/#data-protection-and-review) lists what remains your responsibility.

### How a Local Model Runs

A local model is a chain of four parts:

1. **A GGUF file** holds the weights and metadata of the model.
2. **llama.cpp** is the native inference engine. It runs the model in a separate `llama-server` process.
3. **TXTextControl.AI** loads the model, starts and owns the process and creates chat sessions.
4. **`IChatClient`** of `Microsoft.Extensions.AI` is the standard .NET interface through which your application talks to the model.

`TXTextControl.AI` is the package your application references; `TXTextControl.AI.LlamaServer`, which manages the native process, the runtime installation and hardware reporting, comes with it. Neither package contains model weights, the native `llama-server` executable or GPU drivers, and restoring the packages or starting the application doesn't download a model.

### Provide the Runtime

The native engine is provisioned separately from the packages. There are three ways:

- **Install it once.** `LocalRuntime.InstallAsync` downloads the pinned llama.cpp release, verifies its size and SHA-256 hash, extracts it and tests it. This is a setup step, not something to do at every application start.
- **Use an existing executable.** Set `LocalModelOptions.LlamaServerExecutablePath` to its absolute path, or the environment variable `TXTEXTCONTROL_AI_LLAMA_SERVER`.
- **Allow an automatic download.** With `RuntimeAcquisition = RuntimeAcquisitionPolicy.DownloadIfMissing`, loading a model installs a missing runtime. The default, `NeverDownload`, never accesses the network for a runtime.

The managed installation uses a fixed llama.cpp release, `b10621`, downloaded from the llama.cpp releases on GitHub:

| Platform | Backends that can be installed |
|---|---|
| Windows x64 | CPU, CUDA, Vulkan |
| Linux x64 with glibc | CPU, Vulkan |
| macOS and Linux with CUDA, musl or ARM64 | None: supply an executable or an external endpoint |

On Linux, the runtime needs `libgomp.so.1`, from the `libgomp1` package on Ubuntu, even for CPU inference. Installed runtimes are cached under the local application data folder of the running account, for example *%LOCALAPPDATA%\\TXTextControl.AI\\runtimes\\llama.cpp* on Windows; for services and containers, set `RuntimeCacheDirectory` to a persistent, writable folder. To download from an internal mirror instead of GitHub, set `RuntimeDownload` to a `RuntimeDownloadOptions` object with the mirror's `BaseUrl`; the mirror must serve identical files. Once the runtime and the model are in place, inference runs offline.

### Load a Model

Models are GGUF files that you obtain separately, in compliance with their licenses. The blog article on running a local LLM uses *Qwen\_Qwen3.5-4B-Q4\_K\_M.gguf*, a quantized GGUF conversion of Qwen3.5-4B downloaded from Hugging Face, of about 3 GB. The following console program installs the runtime, reports what it found, loads the model and asks a question:

```csharp
using Microsoft.Extensions.AI;
using TXTextControl.AI;

LocalModelOptions options = new LocalModelOptions
{
    HardwareBackend = HardwareBackend.Cpu,
    ContextSize = 8192,
    MaxOutputTokens = 1024
};

// Setup step: check the machine without network access, then install the runtime
var status = await LocalRuntime.InspectAsync(options);
Console.WriteLine($"{status.Platform}, llama.cpp {status.Version}, recommended backend: {status.RecommendedBackend}");

if (status.InstallationBlocker is not null)
{
    Console.WriteLine(status.InstallationBlocker);
    return;
}

await LocalRuntime.InstallAsync(options);

// Load the model once and reuse it
await using var model = await LocalLanguageModel.LoadAsync(
    Path.GetFullPath("Models/Qwen_Qwen3.5-4B-Q4_K_M.gguf"), options);

Console.WriteLine($"Model: {model.Info.Name}, runtime: {model.Runtime.Engine} / {model.Runtime.Backend}");

using var session = model.CreateSession();
var response = await session.GetResponseAsync("Explain a document approval workflow.");
Console.WriteLine(response.Text);
```

The code works as follows:

- `LocalRuntime.InspectAsync` reports the platform, the pinned version and a recommended hardware backend without downloading anything. `InstallationBlocker` names a missing requirement, such as `libgomp.so.1` on Linux.
- `LocalRuntime.InstallAsync` installs the runtime. Later runs reuse it, so a production application loads with the default `NeverDownload` and skips this step.
- `LocalLanguageModel.LoadAsync` starts the `llama-server` process with the model. Relative paths are resolved against the application folder and then the working directory, so an absolute path avoids surprises. Disposing the model stops the process.
- `model.Info` describes the model, and `model.Runtime` the engine and the backend in use.
- `CreateSession` returns a chat session, an `IChatClient`. Load a model once and create sessions from it, instead of loading it for every request.

The most important `LocalModelOptions` and their defaults:

| Option | Default | Sets |
|---|---|---|
| `HardwareBackend` | `Auto` | `Auto`, `Cpu`, `Cuda` or `Vulkan`; `Auto` prefers an available accelerator and falls back to the CPU |
| `ContextSize` | `8192` | The tokens of one request, input and output together |
| `MaxOutputTokens` | `1024` | The maximum length of an answer |
| `GpuLayers` | `-1` | The number of model layers offloaded to the GPU; `-1` offloads as many as possible |
| `Temperature` | `0.6` | The randomness of the answers, from 0 to 2 |
| `RuntimeAcquisition` | `NeverDownload` | Whether loading may download a missing runtime |
| `LlamaServerExecutablePath` | `null` | The absolute path of an existing `llama-server` executable |

### Choose Hardware and Model Size

- **The file size isn't the memory requirement.** The context cache, working buffers, GPU offloading, other applications and concurrent requests need memory, too.
- **Start small.** For development with small models, a modern multicore CPU, an SSD and 16 to 32 GB of RAM are a practical starting range, with the upper end when the Document Editor, the MCP document server, a development environment and embeddings run together. These are planning suggestions, not certified requirements.
- **CPU inference works**, but measure the latency for your task. Use a backend and driver combination that you have validated with your GPU and your model; on Windows x64 with a compatible NVIDIA GPU and driver, set `HardwareBackend` to `Cuda`.
- **Increase the context step by step.** Start with a context of 8,192 tokens and increase it after measuring memory use and how much of your documents a request covers. Don't raise it beyond the memory or the context the model supports.

Tool calling, structured output and complex instructions depend on the capabilities of the model and runtime. A successful load says nothing about the quality of the answers, so test a model with your own documents and tasks before you choose it.

### Where Inference Runs

- **In your application.** `LocalLanguageModel.LoadAsync` starts and owns the inference process on the same machine, as in the program above.
- **On an existing llama-server.** `LocalModelOptions.LlamaServerEndpoint` connects to a `llama-server` that runs elsewhere, for example on a GPU server, with `LlamaServerModelId` and `LlamaServerApiKey`. Protect that endpoint with TLS and authentication, and don't expose the raw inference port.
- **In a separate AI service.** The `AiService` sample hosts inference, runtime management, embeddings and knowledge collections independently of the website, which forwards requests over an authenticated connection. Several departments can share one controlled model deployment this way; it's not an automatic multi-GPU cluster, so run a single service instance.

The samples repository [txtextcontrol-ai-samples](https://github.com/TextControl/txtextcontrol-ai-samples) describes the remote service, runtime administration and download mirrors in its *docs* folder.

### Hosted Models as an Alternative

Since version 0.1.0-beta.2, `TXTextControl.AI` can also connect to OpenAI and OpenAI-compatible APIs with `OpenAIChatClientFactory`. Local inference remains the default, and there is no automatic fallback to a cloud service. A hosted model receives the prompts and the document text included in them, so the benefits of local processing described above don't apply to it. Older blog articles, such as [PDF Document Classification with OpenAI and TX Text Control in C#](https://www.textcontrol.com/blog/2024/01/26/pdf-document-classification-with-openai-and-tx-text-control-in-csharp/), send document text to OpenAI's hosted API.

The blog article [Running a Local LLM in .NET C# with TX Text Control AI](https://www.textcontrol.com/blog/2026/09/10/running-a-local-llm-in-dot-net-c-sharp-with-tx-text-control-ai/) builds an interactive console chat with a local model step by step.

### Next Steps

- [Chat with Documents](https://docs.textcontrol.com/tx-text-control-dotnet-server/ai/chat/llms-full.txt): ask a local model questions about a document.
- [Text Control AI](https://docs.textcontrol.com/tx-text-control-dotnet-server/ai/text-control-ai/llms-full.txt): the packages and samples of the preview.
- [RAG](https://docs.textcontrol.com/tx-text-control-dotnet-server/ai/rag/llms-full.txt): answer questions from a collection of approved documents.

---

## Nearby

- **Previous:** [Text Control AI](https://docs.textcontrol.com/tx-text-control-dotnet-server/ai/text-control-ai/llms-full.txt)
- **Next:** [Chat With Documents](https://docs.textcontrol.com/tx-text-control-dotnet-server/ai/chat/llms-full.txt)