Despite having been in the AI game since before ChatGPT became a household name, the world of local LLMs admittedly once terrified me. What’s interesting here was that it was never the quality that scared me off. Instead, it was everything around it. The setup looked complex, it demanded a fair bit of technical tinkering, and it seemed to require hardware far beefier than anything I owned. Besides, watching my colleagues over at XDA go on and on about self-hosting every possible thing and casually name-dropping terms like Docker, containers, and home servers left me quite convinced it wasn’t for me.
However, the world of local LLMs has gotten a lot more accessible and the barrier I built up in my head no longer matched reality. So, I replaced all the LLM apps I have installed on my phone with a single local model for a week. And while there’s plenty to love about the arrangement, here’s what I had to give up in the process…
Local LLMs can now do a surprising amount on your phone
The barrier was mostly in my head
Before I jump into the tradeoffs that came with this setup, I should be clear about what I was actually running and the benefits. My daily driver here is Gemma 4-E2B-it, a 2.54 GB model I installed through Google’s AI Edge Gallery app and now runs entirely on my iPhone 15 Pro Max. As I mentioned in the introduction, the world of local LLMs has gotten significantly more accessible, and my setup reflects that fairly well. All I needed to do was install the app, and then the model.
Once it’s downloaded, it lives on the device and everything after that happens offline. It’s been impressive enough that I’ve handed it the bulk of my everyday, low-stakes AI use: cleaning up emails, explaining concepts I half-remember from a lecture, breaking down code I’m stuck on, converting units mid-recipe.
I also use it for the private stuff: a message to my travel agent with my passport number in it, a screenshot of a DM I want a second read on, a half-formed question about my own finances, none of which I have to redact anymore because nothing leaves the phone. It even holds its own on quick visual questions through Ask Image and handles transcription through Audio Scribe.
Given that local LLMs work offline too, I rely on them when I’m out of signal entirely, like when I’m on a flight with no Wi-Fi. For all of that, it clears the bar easily. Finally, since local LLMs work entirely on your own hardware and there isn’t a server involved at all, there are no rate limits to worry about. I’ve spent a lot of time and energy complaining about Anthropic’s limits for Claude, and the fact that every trivial query I offload to Gemma is one that never touches my quota has made those limits far less of a daily annoyance.
Local LLMs are no use when the question is about right now
Frozen in time, and it shows
When you download a local LLM onto your device, you essentially install anything and everything it knows and will use to answer. Large language models are trained on enormous sets of data, and that training bakes everything the model learns into what are called weights — billions of numbers that encode the patterns it picked up. Those weights are the entire model, and that’s what you download onto your own device instead of it living on someone else’s server.
Now, all the training data of the model you’re using has a cutoff point, which is a date beyond which the model simply hasn’t seen anything. Everything the model knows is frozen into those weights at that one moment in time. This isn’t just limited to models you run locally. Even Opus 4.8 and Fable 5 and GPT 5.6 have a training cutoff. Every model does, and the difference is what happens next.
When you ask Claude or Gemini about something that happened this morning, it can reach out to the web, pull in live results, and fold them into its answer. A local model on the phone can’t do this by default, and is working purely off what’s baked into those weights. This means the moment my question depends on anything recent, it’s got absolutely nothing to tell me.
A phone-sized model has a phone-sized brain
Ask too much and the seams show
To understand where this ceiling comes from, it helps to think about what you’re actually up against. When you use a cloud model like Claude or Gemini, you’re tapping into something enormous. Those models live on massive server infrastructure, racks upon racks of specialized hardware sitting in data centers, and the versions you and I talk to run into hundreds of billions of parameters. That’s the muscle behind the answers, and it’s the reason a cloud model can hold a huge amount of context in its head at once and reason through hard problems without breaking a sweat.
A 2.54 GB model running on my phone is playing an entirely different game. It has to be small enough to fit on a device I also use for everything else, which means it’s a tiny fraction of the size, and that tradeoff shows up the moment the task gets hard.
For the quick, everyday queries I described earlier, that ceiling never comes into play. However, the moment I hand Gemma something demanding, the gap becomes impossible to ignore. I noticed it most with the kind of work I actually care about. When I feed a cloud model a long, messy document and ask it to reason, or when I’m several layers deep in a bug and need it to reason across a whole chunk of code, the difference in horsepower is obvious. Gemma will give it a genuine attempt, but the answers get shallower the harder the task gets.
None of this is a knock on Gemma or local LLMs as a whole. Those larger cloud models run on all that infrastructure for a reason, and expecting a 2.54 GB download to match them was never realistic. So when the task is genuinely demanding, I still find myself back at my desk with a cloud model, because that’s simply where that kind of work belongs.
You don’t realize how much the ecosystem matters until it’s gone
A model on its own is only half the tool
With a Claude Pro subscription, you get Claude Code, Cowork, Projects within Claude, Artifacts, Design, and a whole suite of connectors that plug the model straight into the tools I already use. With a Google AI Pro subscription, you get Gemini woven through Gmail, Docs, and the rest of Workspace, plus NotebookLM, Deep Research, Google Antigravity, Gemini within Google Search, and more. With a ChatGPT Plus subscription, you get custom GPTs, Codex, Projects, connectors, image creation, scheduled tasks, ChatGPT Work, deep research, memory, and so on.
The common thread here is that almost none of what makes these tools as powerful as they are is limited to just the raw model at the center of them. It’s everything built around it, and the entire ecosystem you get along with it. You’re paying for a tool’s ability to act on your files, to run a task end to end, to plug into the apps where your work already lives, and to remember what you told it last week. That entire layer is exactly what you give up the moment you switch to a model running entirely on your phone.
With a local LLM on a phone, you essentially just get a model in a box by default. It’ll answer whatever I type into that one app, but it can’t touch my inbox, open my calendar, read a document I didn’t paste in by hand, or recall a single thing from yesterday’s conversation. There’s no agent working through a task in the background, no research mode going off to read forty sources for me, nothing reaching into the tools I actually use.
While I can most certainly configure a lot of this myself, wiring the model up to my files, giving it tools to call, building the kind of integrations that come standard with a cloud subscription (or using community-built ones), it’s nowhere near the deep, native integration a cloud subscription gives you the moment you sign in. Everything a cloud subscription gives me by default becomes something I have to assemble and maintain myself, and most people switching to a local model aren’t signing up for that.
I’m not claiming that local LLMs can replace the cloud tools I lean on every day because they really don’t need to. I’ve come to realize that the best setup isn’t one or the other, but knowing which tool to hand each job to. The quick, personal, offline, everyday stuff lives on my phone now, running for free and never leaving the device, while the demanding work that needs real reasoning, live information, or the whole ecosystem wired around the model still goes to the cloud, where I’m happy to keep paying for it when it actually counts!
