I’ve always been a fan of repurposing old hardware as server nodes, and the spike in PC hardware prices has pushed me further down this rabbit hole. With LLMs becoming more usable for productivity tasks, I frequently try to revive aged systems as AI-hosting workstations. In fact, pairing the right model with a handful of optimizations goes a long way in bolstering any outdated machine’s LLM-hosting capabilities.
And I’m not just talking about bulky enterprise-tier hardware, either. I’ve got an Acer Predator Helios 300 (to be precise, a G3-571-77QK variant) with a GTX 1060 (6GB) and 16GB of DDR4 RAM that I bought back in 2017, and despite being a relic of the bygone Pascal era, this gaming laptop is an incredible llama.cpp server and caters to the rest of my containerized productivity stack.
I use my Proxmox-powered laptop to drive my LLM workloads
But you can also use any other Linux distribution and expect similar results
Let me be clear: DietPi, AntiX, Fedora, and even Linux Mint are perfect lightweight server distros for a project of this caliber. However, I ended up going with Proxmox, as I’d already set it up on my old laptop as part of an experiment several months ago, and it has the killer container management (and isolation) features that I need to replace ChatGPT and other cloud apps with a completely local setup.
Since I’ve heard horror stories about folks running into passthrough issues with the latest versions of Proxmox on their GTX 10-series cards, I decided to leave mine at 9.0. Likewise, Nvidia had removed support for Pascal GPUs in recent updates, so I decided to roll with driver version 580. Once I’d installed the drivers on the host, I passed the GPU to the LXC by tweaking the latter’s config files and appending the right permissions, I repeated the driver setup procedure on the LXC (except, I had to add the –no-kernel-module flag).
The process was extremely similar to how I configured a GTX 1080-powered llama.cpp instance on my old gaming PC, but these are three commands I used to pull the right drivers for my laptop’s Pascal card:
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/580.119.02/NVIDIA-Linux-x86_64-580.119.02.run
chmod +x NVIDIA-Linux-x86_64-580.119.02.run
./NVIDIA-Linux-x86_64-580.119.02.run
Setting up llama.cpp within an LXC required a bunch of tweaks
While the passthrough aspect was fairly simple, I had to put a lot of elbow grease into ensuring llama.cpp played with my GPU. When I’d tried to run LLMs on an LXC connected to my dinosaur GTX 1080 a few weeks ago, I had to compile a Vulkan-based version of llama.cpp. Otherwise, the inference engine would refuse to detect my GPU and instead switch to CPU for the processing tasks.
So, I decided to go down the Vulkan route for my GTX 1060-powered pipeline as well. First, I ran the apt install libvulkan1 vulkan-tools command on the host to grab the Vulkan packages before executing vulkaninfo | grep -i deviceName to confirm everything works as intended.
Then, I switched to the LXC and ran apt install glslc glslang-tools libvulkan1 vulkan-tools libvulkan-dev spirv-tools spirv-headers build-essential git cmake curl libxext6 to install every pre-requisite package needed by llama.cpp. I also had to ensure that the Vulkan drivers worked on my GPU within this LXC, so I executed nano /usr/share/vulkan/icd.d/nvidia_icd.json and appended the following code (with the correct indentation rules) to it:
{
"file_format_version" : "1.0.0",
"ICD": {
"library_path": "libGLX_nvidia.so.0",
"api_version" : "1.3"
}
}Running vulkaninfo | grep -i deviceName confirmed that my GTX 1060 was a potential Vulkan device, so I pulled the llama.cpp files via git clone https://github.com/ggerganov/llama.cpp and switched to its folder with cd llama.cpp. Finally, I ran cmake -B build -DGGML_VULKAN=ON followed by cmake –build build –config Release -j$(nproc) to compile llama.cpp from scratch.
The tiny Gemma 4 E4B meshes well with my laptop’s GTX 1060
It crushes most workloads with ease
Considering the sheer number of small models that fit inside a 6GB VRAM graphics card, I had plenty of options at my disposal. Since I wanted decent response times without the accuracy drawbacks of running sub-2B LLMs, I went with Gemma 4 E4B in the end. I already had the LLM’s GGUF files on my PC, so I quickly transferred them to my llama.cpp container over SCP instead of downloading them again.
Then, I navigated to the freshly-compiled /build/bin directory within my llama.cpp folder and executed ./llama-server -m “/root/models/gemma-4-E4B-it-Q4_K_M.gguf” –mmproj “/root/models/mmproj-gemma-4-E4B-it-BF16.gguf” -c 100000 –host 0.0.0.0 –port 8082 -ngl 999 –webui-mcp-proxy to get Gemma 4 E4B up and running, with the –ngl 999 forcing the inference engine to use the GPU, –mmproj arming it with vision capabilities (and another GGUF file) and –webui-mcp-proxy letting me use MCP servers with it.
Speed-wise, I was pleasantly surprised to see this LLM clobbering away at 30+ tokens/second.
Sure, its token generation speeds are one-third of what I’m used to on my RTX 3080 Ti, but they’re nothing to scoff at, either. Even if I were to pair my GTX 1060 + Gemma 4 E4B setup with Home Assistant, it’d be responsive enough to answer my queries in real-time. I’ve previously tested Gemma 4 E4B for everything from image recognition and summarization to RAG (with nomic-embed-text providing the embedding capabilities) searches, and it holds up surprisingly well for each task.
Trading ChatGPT for a local LLM added some much-needed privacy to my productivity tasks
If you’re wondering why I’m willing to take so many measures to replace ChatGPT with a self-hosted model, it all boils down to privacy. Sure, ChatGPT may have free tiers, but I’d rather not deal with the privacy implications of using an LLM that runs on some company’s servers to perform inference tasks on my data.
For example, I’ve paired Gemma 4 E4B with my Paperless-ngx pipeline, where Paperless-GPT handles all the OCR tasks and Paperless AI provides contextual chat and tagging tools. Besides the pay-as-you-go pricing of ChatGPT’s API, tossing my bank documents, ID cards, or academic records into this setup would be a privacy nightmare. To a lesser extent, this also applies to my Open Notebook research, Karakeep bookmarks, Open WebUI voice assistant, and Blinko notes, as I wouldn’t want ChatGPT to keep tabs on my productivity workflows.
