Choosing the right locally-deployed large language model can be a bit of a hassle. Bulky LLMs, for example, can deliver accurate results, but you’ll need decent hardware to get them up and running. I use Mixture-of-Experts models for most of my LLM-heavy tasks, but even with their optimized nature, I can’t use MoE offloading to run 35B clankers on iGPU-laden systems with less than 8GB of memory.
Meanwhile, tiny models that lie in the sub-7B range can fit on weak systems, but the precision trade-off just isn’t worth it. Or at least, that’s what I used to think until I ran Gemma 4 E4B on my Raspberry Pi. After testing it extensively across different systems, I have to admit that this lightweight LLM packs surprisingly high reasoning capabilities for its size.
Gemma 4 E4B can fit inside something as weak as an SBC
But it delivers killer speeds on my GPU-powered workstations
A few weeks ago, I tried running the agent harness Pi on my Raspberry Pi, partly because the alliteration sounded funny, and also since I wanted to gauge the SBC’s performance in LLM-heavy workloads. As such, I tried hosting the AI models I planned to use with Pi on my tiny tinkering companion, which soon devolved into a disaster. Although 1.7-2B models could answer simple questions and run normal terminal commands, they’d either hallucinate like madmen, go into never-ending loops, or have a full-on meltdown when building a usable extension for pairing Pi with the Docker socket on my SBC.
Gemma 4 E4B, however, was the only model that delivered solid results. It automatically found a half-baked extension created by another clanker, modified most of its code, and had zero issues using it to answer my queries. Here’s the fun part: despite possessing as much knowledge as a conventional 8B model, Gemma 4 E4B uses Per-Layer Embeddings to reduce its effective parameters to 4.5B. You see, the PLE architecture grants an embedding table to every decoder layer in Gemma 4 E4B, thereby letting the model access more information without hogging too many system resources.
As a result, Gemma 4 E4B performs exceedingly well on weak systems. Considering that my Raspberry Pi 5 (8GB) has trouble loading 6B (and even certain 5B) models, the fact that Gemma 4 E4B works on the SBC is pretty incredible. Of course, the token generation rate varies between 2.95–3.25 t/s, which isn’t a lot for everyday tasks. So, I’ve tried running it on my MoE model-hosting workstations, and it runs pretty darn well on both systems. My GTX 1080 can run it at 30–40 tokens/second, while my RTX 3080 Ti can drive Gemma 4 E4B at nearly triple that number. And the best part? It supports audio processing and vision capabilities, making it an all-in-one LLM for quick inference tasks.
I’ve even used it to troubleshoot faulty code and server logs
Since I wanted to test Gemma 4 E4B’s reasoning prowess, I paired it with my containerized services that support LLMs. For reference, I ran the llama-server command with the –mmproj, –jinja, and –webui-mcp-proxy flags to extend its processing capabilities even further. First, I asked it to summarize a couple of PDFs directly from the llama-server web UI, and it did a pretty good job at retrieving important information from the documents. I also tossed a couple of images and asked it to describe what they were about, and I have nothing to complain about besides a single instance where the model failed to detect the app running on my laptop (but in all fairness, Kage is a relatively obscure tool). Since I had a Docker MCP server hooked up to llama-server, I used the LLM to execute some management commands. While it wasn’t able to spin up new containers (and I’ll get to that in a bit), it had no trouble pulling new images, checking on the existing Docker environments, or even generating commands for new services.
With Open Notebook and Blinko, Gemma 4 E4B had zero issues summarizing notes and processing RAG queries. In fact, it was able to filter all the unnecessary information out of my sources and answer my queries with relevant details. It generated tags just as well for my Karakeep and Paperless AI containers, and I had decent results when I asked it to look into buggy code and long server logs.
But it’s not powerful enough to replace my bulky MoE models
Complex tasks are a little too much for this tiny LLM
As much as I adore Gemma 4 E4B, I have to admit that I’d rather rely on my 26B and 35B models for complex reasoning tasks. Compared to Qwen3.6-35B-A3B, which manages to spin up fully-functional containers using nothing more than the image name, Gemma 4 E4B fails at calling the container creation tools from the Docker MCP server. Likewise, asking it to create full-on Ansible Playbooks isn’t something I’d recommend, unless you want malformed code with multiple errors.
I also tried using it as the conversation agent for my Home Assistant server, and while the results weren’t terrible by any means, it did fail at selecting the right smart device from my (admittedly) vague commands. It couldn’t create complex automation chains without messing up the if and then conditions, either.
That said, I plan to integrate Gemma 4 E4B into my LLM workflow. I’ve got a spare laptop with the ol’ reliable GTX 1060 that’s gathering dust, and with a little bit of LXC GPU passthrough wizardry, Gemma 4 E4B should turn my aged computing companion into a solid LXC-hosting PVE workstation.
