Local AI: run Large Language Models securely on your own infrastructure
A local LLM is a language model that you run on your own laptop, server, or GPU machine, for example, using Ollama or GPT4All. Questions and documents remain within your own network, and you don’t pay per token. On the other hand, you have to purchase the hardware yourself and manage and update the model on your own.
The use of Large Language Models (LLMs) such as ChatGPT has become hugely popular since OpenAI released ChatGPT in November 2022. Since then, many more companies have launched similar LLMs, such as Google’s Gemini, Anthropic’s Claude, xAI (Twitter)’s Grok, and others. While this is good news for users, it also presents a major challenge for organizations: data sovereignty.
It’s great that there are so many options, but choosing one of these providers also means deciding which organization is allowed to process all the data you send when you interact with their chatbot. Choosing a provider often means trusting that they will handle every piece of information you enter with care. For individuals, that privacy consideration might seem minor when it comes to a single query. For businesses, the stakes are much higher. Organizations need to ask themselves one question: Can we take the risk of employees uploading source code, financial spreadsheets, or customer documents to a third-party cloud?
Fortunately, there are ways to harness the power of LLMs without sending all kinds of data to OpenAI and similar providers. In this blog post, I’ll show you how to set up LLMs locally on your laptop, PC, or server, and how to use an LLM privately on your personal documents—all while still enjoying benefits such as lightning-fast summaries, creative brainstorming, and advanced data synthesis. By hosting these models on your own hardware, you gain the freedom to process sensitive datasets—such as financial data or private intellectual property—without worrying that they’ll be used for training or stored on an external server. This bridges the gap between state-of-the-art intelligence and uncompromising data sovereignty.
Before I go into the setup and what you need, I will first provide some background information about LLMs and terms that you often encounter online. The installation itself is relatively simple, but if you delve into it, you will quickly see that there are many variants: model types, file formats, parameter sizes, quantization methods, and so on.
Model Files
There is a wide variety of public LLM models available. A popular place to find models is huggingface.com/models. People and companies upload their models there, although the website isn’t always very intuitive. Many tools also offer an easy way to download models. Finding free models is easy; choosing a suitable model or file requires a bit more attention. Popular free models to explore include Phi (Microsoft, smaller models), Qwen (Alibaba), Llama (Meta), Mistral, DeepSeek, and Gemma (Google). Be sure to try out a few popular models and see what works best for your use case.
File types
Most LLM models you find online are .gguf or .safetensors files. These files contain the complete LLM model and can therefore be quite large. Make sure you use a tool that supports one of these file formats, otherwise you will limit your choice and ultimately the usability of your local deployment.
The role of an LLM
Many LLMs are trained for a specific role or use case. This is often reflected in the file name. If it contains "instruct," the model is intended for chat-like interaction, such as ChatGPT. If you see "text" or no clear indication, it is usually a base model, primarily intended to continue (complete) text. You may also encounter terms such as "reason(ing)" or an "r" version: similar to instruct, but the LLM also displays the reasoning steps it took to arrive at the answer.
Parameter size and quantization
When selecting a model, you’ll often see multiple variants of the same model. For example, if you want to use qwen2.5, you can choose from 0.5b, 1.5b, 3b, 7b, 14b, 32b, and 72b. This number represents the number of parameters (b = billion). The higher the number, the larger the model, and generally, the better it is at handling complex tasks.
But file sizes are also growing rapidly, and with them, the RAM/VRAM requirements needed to load the model. Because files can become so large, there are methods for reducing the model’s size with as little loss of quality as possible: quantization. In the image below, for example, you can see the various quantized versions of Qwen2.5 32b.
You often choose quantization because your hardware is limited, but you still want more parameters. Keep in mind that this usually comes at the expense of quality. Keep it practical: consider the file size and what fits in your memory. More on that later.
What do you need?
You can run LLMs on virtually any computer, even on embedded devices such as a Raspberry Pi or a smart camera. However, the performance and the types of models you can use are limited. The two most important hardware components to consider are CPU RAM and GPU VRAM.
GPUs, which are best suited for running and training LLMs, have a limited amount of working memory. For maximum performance (and, increasingly, just to be able to run a model at all), one or more dedicated GPUs are required, with NVIDIA typically offering the best support.
Due to the matrix calculations required by LLMs, you can also run them on CPUs, but performance is usually significantly lower than on GPUs, which are optimized for these types of calculations.
To determine how much (V)RAM you need, you can roughly estimate based on the model’s file size.
There are tricks to improve CPU performance, but that is beyond the scope of this article.
Note: It is technically possible to load larger models that do not fit in your RAM; the system will then swap to your SSD/HDD. However, this makes the LLM extremely slow and does not provide a pleasant user experience.
Setup
For this example, we'll use qwen3:8b-q4_K_M. My test system has a 12-core CPU and 32 GB of RAM and runs on Linux, without a dedicated GPU.
The first tool I use is Ollama. Ollama is a CLI interface that allows you to interact with an LLM. It also unlocks the LLM via HTTP, allowing you to access it via curl or via Python and JavaScript. To install Ollama, go to https://ollama.com/download and select your operating system. After installation, run the command ollama run qwen3:8b-q4_K_M. This automatically downloads the model and starts it, after which you can ask questions. Done.
You now have a fully functional LLM running locally, without sending data to a data center. The image below shows an example in which qwen3 is a thinking model, meaning it reasoned about the steps needed to provide an answer.
Ollama is ideal for quickly starting an LLM and when you want to build an LLM agent in Python or JavaScript. You can also combine it with LangChain/LangGraph. Ollama also supports embeddings (for documents), but this requires some programming work. There are also tools with a UI that give you a ChatGPT-like experience in the browser.
GPT4All
GPT4All offers a user interface that lets you easily install models, create custom databases with information the LLM can use, and access a chat interface that clearly displays the conversation. Please note: GPT4All collects anonymous user data, but you can disable this in the settings or run it completely offline. Its strength lies in the fact that you can easily upload local documents and then ask the LLM questions about those documents.
Download GPT4All from https://www.nomic.ai/gpt4all. After launching the app, go to “Chat” or “Models” and install a model. It will show you how much RAM you need. In this example, I chose DeepSeek-R1-Distill-Qwen-14B. Then I went to the “LocalDocs” tab and created my own document store. I selected a folder containing one PDF: the basic rule set for Dungeons and Dragons, 115 pages.
GPT4All then creates embeddings—in other words, it converts words from documents into vectors so that the LLM can use the information in the chat. Depending on the number and size of the documents, embedding can take a while; this also depends on your system. By default, GPT4All only embeds files with the following extensions: .docx, .pdf, .txt, .md, .rst. You can add more extensions, but GPT4All cannot guarantee that it will always work properly.
Then go to Chats, select the model you downloaded, and choose the LocalDocs you just created in the top right corner. Simply ask a question about the documents and the model will use the documents to provide answers. This may be a bit slow, so please be patient. I asked DeepSeek: "What does the ruleset say about Duergar?" The answer can be seen in the image below and corresponds to the description of the Duergar in the ruleset.
Why would you use a local LLM?
Data Protection
As mentioned in the introduction: the strength of a local LLM is that all the questions you ask and all the documents you upload remain within your local environment. No data is shared with a data center. Furthermore, the model itself is not capable of “storing” the data you enter.
This means you have complete control over your data and can safely ask questions about proprietary source code or confidential documents.
This is also valuable in situations where a system must be air-gapped (i.e., disconnected from the internet). Both Ollama and GPT4All keep data within your local network and do not rely on an internet connection.
Cost Savings
By running locally, you avoid API costs, which can quickly add up with high volumes and constant use. You do have initial investment costs and operational costs (such as electricity), but it doesn’t matter whether you generate 10 tokens or 10 million tokens: you don’t pay per token. You’re moving from variable, unpredictable API costs to a fixed investment.
An added benefit: paid LLM APIs often have rate limits on the number of requests. This can cause downtime or incur additional costs to increase your rate limit. Locally, there is no rate limit on the API; your hardware is the only limitation.
Both Ollama and GPT4All can make models available through their own APIs.
This is useful if you already have applications that query LLMs via an API. It allows you to reduce the costs you would otherwise incur by using paid LLM APIs.
Latency
With GPUs, the latency of a locally hosted LLM can be significantly lower than with API calls to a data center. You are communicating with a local system that is available to you, without a long network route and without shared resources. This increases the response speed.
Trade-offs
Of course, there are also disadvantages to self-hosting. You have to manage the LLM model, keep track of updates, and sometimes do troubleshooting. That takes time and effort and is therefore part of the TCO. In addition, hardware costs (CAPEX) can be a significant initial investment, rather than a pay-as-you-go (OPEX) model.
Ollama requires a bit more setup but offers a lot of flexibility. GPT4All is more of an all-in-one package with a user interface and makes uploading documents very easy, but it offers less flexibility. For example, if you want to offer employees a ChatGPT-like experience—but locally—then GPT4All is often a good choice. If you have applications that need a generic interface with an LLM, then Ollama makes more sense because it offers a general API that many tools can work with.
Do you need a private LLM for your business?
A private LLM is worthwhile when employees ask questions about source code, financial spreadsheets, or customer documents that cannot be shared with OpenAI or any other external provider. For systems that must run in an air-gapped environment, a local solution is also the only option: Ollama and GPT4All work without an internet connection.
If you’re mainly asking general questions and don’t want to invest in GPUs and management, a paid API is the easier way to get started. You’ll pay per use rather than upfront for hardware. If you choose an on-premises solution, your use case will determine which tool to use. If you want to provide employees with a ChatGPT-like environment, GPT4All is the obvious choice. If applications need to interact with the model via an API, Ollama is a better fit.
Get started with local LLMs today
The shift toward local AI represents a fundamental change in how we interact with technology: from “data in exchange for services” to true digital sovereignty. As open-source models continue to close the gap with closed-source alternatives, the case for “hosting your own AI” grows stronger. If you start your local LLM journey today, you’re not only protecting your data; you’re also building a private foundation for the future of your digital workflow.
If your organization is exploring how to use AI safely and responsibly, we can help you evaluate your options and design a setup that balances performance, compliance, and long-term costs.
To learn how SUE secures cloud environments, visit the Cloud Security page. The Cloud Security Guide explains how to set up and maintain a secure cloud environment.
