What you actually need to start
The most common mistake is buying hardware before deciding what you want to do. The model determines the hardware, not the other way around. Here's the decision tree:
| Use case | Model size | Hardware | Latency target |
|---|---|---|---|
| Drafting / summarization | 3B–9B | Modern laptop, 16 GB RAM | ✓ <1s/token (fine) |
| Coding / agentic work | 12B–32B | Desktop GPU 24GB VRAM | ✓ <1s/token |
| Hard reasoning / planning | 33B–70B | 80GB VRAM / multi-GPU | 2–4s/token (acceptable) |
| Fine-tuning / RAG training | 7B–13B | 24GB+ VRAM, lots of RAM | Doesn't matter |
If you're doing drafts, summaries, and light coding, a 12B model on 24GB VRAM is the sweet spot. If you need deep reasoning, you either go bigger (33B+) or offload to a cloud API for the hard parts. Most people are somewhere in between — and that's fine. You don't need to solve everything locally.
The four things that waste time
From building and maintaining local setups, these are the mistakes people keep making:
- Chasing the newest model. Qwen 3, Llama 4, Mistral — every week, something "better" drops. In practice the difference between the top 5 models at the same size class is small. Pick one, learn its quirks, and move on. You'll get more value from understanding your model than from swapping it monthly.
- Over-investing in the UI layer. There's no point in polishing a web interface on top of a model that can't reason. Get the model right first. The interface can be as simple as a terminal prompt and a text file. The Hermes desktop app is one example — but a plain terminal + Ollama API works for 80% of use cases.
- Treating local models as a replacement for cloud. Local models are slower and weaker on hard tasks. The right architecture is: local for drafts, summarization, and privacy-sensitive work. Cloud for the reasoning, planning, and hard coding tasks. Running both is not overengineering — it's the correct setup.
- Ignoring context window management. Models have a limit on how much text they can process. If your input exceeds that, you get truncated or low-quality output. For anything above ~10k tokens of context, you need RAG (retrieval augmented generation) — a simple vector store + a small model that retrieves relevant chunks. Don't skip this step.
A working setup, step by step
This is a real setup that runs a newsletter pipeline, does agentic coding, and handles voice notes. It's not the most powerful setup — it's the one that's actually being used and maintained.
A Telegram message arrives: "Summarize this YouTube video and write a newsletter section about it." The local model takes a transcript from the knowledge base, runs a summarization pass, writes a draft, and the draft is handed to a stronger model for the final polish. Total time: a couple of seconds locally, under two minutes end to end.
A concrete example: an automated newsletter
The clearest way to see an agent system is to watch one doing a real job end to end. Consider a newsletter that turns a pile of video content into a weekly, edited, deliverable email. Here is the pipeline in the order it actually runs — each step is one agent call or one script, and a human approves before anything leaves the building:
It uses every idea from this page — local model, vector store, deterministic glue, a human in the loop — and it has a finished output you can see. A newsletter is not a toy; it is the smallest honest version of an agent doing a real job. If you can build this, you have built the thing that every bigger agent system is made of: gather → index → draft → assemble → human-check → deliver.
Bonus: you don't need to be a programmer
The most common reason people never build a local agent is the belief that they need to be a developer. That is no longer true for the 80% of useful setups, and here is why — because the skill is in the arrangement, not the code.
- The model isn't yours to build. A big model (local, via a runtime like Ollama, or a cloud API) ships as a download and a config. You are choosing and connecting, not training. Training is the 1% that actually needs a data-science degree.
- The glue is one file, not a codebase. A working RAG loop — "read a text, split it, store it, retrieve the relevant chunk, hand it to the model" — is about thirty lines. It is a recipe you write once and re-use. That is a scripting task, not a software-engineering task.
- The schedule is a cron line, not a build system. "Run this every Monday at 6am" is a one-line scheduled task on any machine. Timers, retries, and error emails are configuration, not architecture.
- The interface exists — you don't have to ship an app. A message in a messenger, a web page, a file in a folder: those are the surfaces. The agent talks to them through standard tools (web, files, browser, mail). You point it at them; you don't build them.
The real prerequisites are: a machine that can run a 7B–13B model (a decent laptop or a used GPU box), a few hours of willingness to read documentation, and the discipline to do the boring steps in order — model first, then retrieval, then the glue, then the schedule. Nobody who ships an agent system in a month did it to impress an interviewer. They did it because they had a recurring, boring, repetitive task, and they refused to keep doing it by hand. That last one is the whole secret — and it is a judgment call, not a coding one.
What NOT to do
- Don't fine-tune your first model. Fine-tuning is a solution to a specific problem. If your model is making the same types of mistakes repeatedly and you have a clean dataset of 1000+ examples, yes. Otherwise: better prompting, better context, or a bigger model.
- Don't build a custom RAG pipeline from scratch. LangChain is overkill for most use cases. A simple script that chunks text and stores it in ChromaDB is 30 lines of code and works fine. You don't need a framework.
- Don't run 70B models for drafting. They're 5x slower and only 1.2x better than a 12B model for the same task. Run 70B for the 5% of tasks where it actually matters.
- Don't build a "smart" prompt system. The prompt is not the product. The product is the output. Spend your time on the system around the model (tools, context, verification) and the model will do its job. Prompt engineering is a skill for the 10% of cases where the model needs specific formatting — not a substitute for a good system.
Cost comparison: local vs cloud
Next steps
Learning content. This is not investment advice. Local AI setups vary widely in cost and capability depending on hardware and model choice. The setup described here is a working reference, not a universal recommendation. Sourced from Raoul Pal, Anthony Pompliano, Jordi Visser, Peter Diamandis & Bankless. The World is Spinning Faster Than Before is an educational project.