Run a local LLM with Ollama to cut your API costs

You have 500 emails to sort into action-needed, fyi and spam, and a short script could do it in minutes. Then you remember that every one of those 500 calls goes to a cloud model that charges by the token, so you never write the script. Every classification, every summary, every boilerplate draft burns tokens on a cloud model that’s overqualified for the job.

The fix is straightforward: run a smaller model on your own machine for the mechanical stuff, and save the cloud API budget for work that actually requires reasoning. A model of 4 to 12 billion parameters running locally handles summarisation, classification, data extraction, and first drafts surprisingly well, and costs nothing per token after you’ve downloaded it.

Which of your tasks can you hand to that model, how do you set it up with Ollama, and which tasks stay on the cloud?

Part 1 of 4 Learn

Install Ollama

Ollama wraps the complexity of running local models into a few simple commands. There are no Python environments, no dependency management and no CUDA configuration to set up.

macOS (Homebrew):

brew install ollama

Linux:

curl -fsSL https://ollama.com/install.sh | sh

Windows: Download the installer from ollama.com.

Once installed, start the Ollama service:

ollama serve

If you used Homebrew, run ollama serve in a separate terminal window, or brew services start ollama to keep it running in the background. On Linux, the install script sets Ollama up as a systemd service, so it usually starts on its own.

You can verify it’s running by asking its local web service, which listens on port 11434 of your own computer, for the list of installed models:

curl http://localhost:11434/api/tags

You should see a JSON response, probably an empty model list, since you haven’t pulled anything yet.

Pull your first model

Ollama uses a Docker-like pull model: you download a model once, and it’s available locally from then on.

For a solid starting point:

ollama pull qwen3.5:4b

This downloads Qwen 3.5 at the 4B parameter size, about a 3.3GB download. It’s a general-purpose model small enough for a machine with 8GB of RAM.

Once the download finishes, test it:

ollama run qwen3.5:4b

This opens an interactive chat: type a question, get a response, and type /bye to exit. The first response takes a moment while the model loads into memory, but subsequent responses are fast.

Picking the right model for the job

Bigger models give better answers, and the amount of RAM in your computer limits how big a model you can run. The three models below suit three amounts of RAM. The table will date, because new models come out often:

Sizes are from the Ollama library pages for Qwen 3.5 and Gemma 4, checked 5 November 2026. When you read this later, open the library, pick a current model in the same size range, and check that the model’s download size fits in your RAM with room left for everything else you run: the model’s download size has to fit in your RAM with room left for everything else you run.

Start with the smallest model that fits. If you have 16GB of RAM and want better output, pull the next size up.

ollama pull qwen3.5:9b
ollama pull gemma4:12b

You can have multiple models installed simultaneously and switch between them.

---

Know what goes where

A common mistake is to use a local model for everything, get disappointed by the quality and go back to cloud-only. Sending each task to the right place avoids that.

Route locally: mechanical tasks with clear inputs and outputs:

• Summarising content (meeting notes, articles, research)
• Classifying or categorising items (emails, support tickets, data)
• Writing initial drafts of boilerplate (docstrings, README sections, form emails)
• Extracting structured data from unstructured text
• Text transformation (formatting changes, style conversion)
• Simple code generation with well-defined specs
• Private material you’d rather not send to a cloud provider, such as personal notes

Keep on the cloud: tasks that need reasoning or judgment:

• Complex multi-step reasoning
• Security reviews and anything where a mistake is costly
• Tasks that require calling other tools in a chain
• Voice and style judgment (your brand voice, editorial decisions)
• Novel problem-solving where you need the model’s full capability
• Architecture decisions and code review

Send a task to a local model when you can write a clear, specific prompt and the output is either right or wrong, with no subjective quality gradient. If you need the model to think, keep it on the cloud.

Made-up jobs for illustration. No AI is running on this page.

Local or cloud?

How to read this

Eight jobs you might hand to an AI model. Decide where each one goes, then read why.

Send a task to a local model when you can write a clear, specific prompt and the output is either right or wrong. If you need the model to think, keep it on the cloud.

1. Tag 400 support tickets into five categories

   LocalCloud

   Local. Each ticket gets a clear prompt and an answer that is either right or wrong. Test it on 20 tickets you have already tagged before you switch. Local, says the guide. Each ticket gets a clear prompt and an answer that is either right or wrong. Test it on 20 tickets you have already tagged before you switch.

2. Review the login code for security holes

   LocalCloud

   Cloud. A mistake is costly here, and the review needs real reasoning. Cloud, says the guide. A mistake is costly here, and the review needs real reasoning.

3. Summarise this week’s meeting notes

   LocalCloud

   Local. Summarising is a mechanical job with a clear input and a clear output. Local, says the guide. Summarising is a mechanical job with a clear input and a clear output.

4. Write this month’s newsletter in my voice

   LocalCloud

   Cloud. Writing in your own voice is a style judgement, and small local models struggle with anything that has no single right answer. Cloud, says the guide. Writing in your own voice is a style judgement, and small local models struggle with anything that has no single right answer.

5. Pull the supplier, date and total out of 200 invoices

   LocalCloud

   Local. Extracting fields from text is mechanical, and each field is either right or wrong. Local, says the guide. Extracting fields from text is mechanical, and each field is either right or wrong.

6. Plan the launch of a new product

   LocalCloud

   Cloud. A launch plan takes complex, multi-step reasoning, so you want the model’s full capability. Cloud, says the guide. A launch plan takes complex, multi-step reasoning, so you want the model’s full capability.

7. Turn my private notes into a tidy list

   LocalCloud

   Local. Reformatting text is mechanical, and private material can stay on your own computer. Local, says the guide. Reformatting text is mechanical, and private material can stay on your own computer.

8. Look up a customer, check their orders and draft a reply

   LocalCloud

   Cloud. The model has to choose tools and call them in a chain, and small local models are unreliable at that. Cloud, says the guide. The model has to choose tools and call them in a chain, and small local models are unreliable at that.

Nothing sorted yet.

Start over

This page sends nothing anywhere. Your answers stay in your browser.

How to read this

A local model runs on your own computer and costs nothing to use once you have downloaded it. A cloud model is bigger and charges you each time you use it.

Choose Local or Cloud for each job. The reason appears underneath, and the guide’s answer is outlined in red. Once you have chosen, the answer stays.

The jobs are written by hand to show the idea. They are not output from a live model.

Got it

Put it to work

Ollama runs a small web service on your computer at http://localhost:11434, and it accepts the same requests as OpenAI’s API. Any tool that lets you change the web address it sends requests to can use it (Ollama’s OpenAI compatibility docs, checked 5 November 2026).

Here’s a basic example using curl:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5:4b",
    "messages": [
      {
        "role": "user",
        "content": "Classify this email as: action-needed, fyi, or spam.\n\nSubject: Q3 budget review meeting\nBody: Hi team, please review the attached budget before Friday."
      }
    ]
  }'

The response format matches OpenAI’s API, so existing code that talks to GPT models works with minimal changes: swap the base URL and model name.

For a more practical pattern, here’s a simple bash function you can drop into your scripts:

local_llm() {
  jq -n --arg p "$1" '{model: "qwen3.5:4b", messages: [{role: "user", content: $p}]}' |
    curl -s http://localhost:11434/v1/chat/completions \
      -H "Content-Type: application/json" -d @- |
    jq -r '.choices[0].message.content'
}

# Usage:
local_llm "Summarise this in one sentence: $(cat meeting-notes.txt)"

The first jq builds the request, so quotes, line breaks and backslashes in your notes can’t break it. The last one pulls the reply text out of the response.

The draft-first pattern

One workflow to try is to let the local model write the first draft of something, then hand it to a cloud model for refinement. The local model does the grunt work of getting words on the page, and the cloud model applies judgment and polish.

This suits boilerplate such as commit messages, docstrings and form emails. Prose that needs your voice starts in the cloud. The local draft gives the cloud model something to react to, which is faster and cheaper than asking the cloud model to generate from scratch.

What to expect

A small model answers short prompts quickly once it has loaded. There’s no network latency, no queue and no rate limiting, so for tasks like classification, where you’re working through a batch of items, the speed difference is big. The first reply is slower, because the model has to load into memory first.

Quality has a ceiling. Local models handle well-defined tasks competently, but they struggle with anything ambiguous or requiring broad knowledge, which is why prose in your own voice stays on the cloud.

A classification prompt that works well on a cloud model may need more explicit examples to get the same accuracy from a small local model. Expect to spend a bit more time on prompt engineering for local models, because being specific pays off more when the model is smaller.

Memory matters, too. While a model is loaded, it occupies RAM, so running qwen3.5:9b next to your IDE, a browser with 40 tabs and Docker containers can feel tight on a 16GB machine. Ollama unloads models after five minutes of inactivity by default (Ollama FAQ, checked 5 November 2026), so this is mostly an issue during active use.

You won’t see a dramatic drop in your API bill overnight, because the expensive calls are usually the complex reasoning tasks you’re still sending to the cloud. What changes is that you stop hesitating to automate small tasks, because the per-call cost is zero: that classification script you wouldn’t write because it would process 500 items at, say, $0.02 each (an illustrative price) is now free to run. The cumulative effect of removing that cost friction adds up over weeks.

Limitations worth knowing

Local models are much smaller than frontier cloud models, and the bigger ones need far more memory: in Ollama’s library, Qwen 3.5’s 27B model is a 17 to 20GB download and its 122B model is 81GB. For the mechanical tasks we’re routing here, that gap doesn’t matter much, but it’s worth keeping in mind if you’re tempted to push the boundary of what “mechanical” means.

Small local models are unreliable at choosing tools and chaining calls. Ollama supports tool calling for some models, but not reliably. If your workflow involves the model deciding which tools to use and calling them in sequence, keep it on the cloud.

Ollama is built for one person’s machine. It can handle a few requests at once, but if you’re thinking about sharing a local model across a team, you’ll want to look at something more production-oriented. For personal workflow automation, Ollama is the right tool.

Part 2 of 4 Practise

Try it on your own work

1. Install Ollama and check it runs. Run curl http://localhost:11434/api/tags.Result a JSON reply with an empty model list.

2. Pull a model. Run ollama pull qwen3.5:4b, then ollama list.Result qwen3.5:4b appears in the list at about 3.3GB.

3. Test it with a real prompt. Run ollama run qwen3.5:4b, paste a classification prompt like the email example with one of your own non-sensitive emails, then type /bye.Result the reply names one of your three labels.

4. Sort your own tasks. Write down five jobs you gave a cloud model this month. For each, ask whether you can write a clear prompt whose output is either right or wrong.Result each job is marked “local” or “cloud”, with the reason beside it.

5. Run the script on a real file. Add the local_llm function to your shell, then run it on a text file of your own.Result a one-sentence summary prints in your terminal.

Part 3 of 4 Check

Check yourself

You tag 400 support tickets into five categories every week, and each one costs a cloud call. Local or cloud, and what do you do before switching?

Local, because each ticket has a clear prompt and an answer that is either right or wrong. Before switching, run the local model on 20 tickets you have already tagged yourself and compare. If it misses, add explicit examples to the prompt.

You want a security review of your authentication code. Do you send it to the local model?

No. A mistake is costly there, and the review needs real reasoning, so it stays on the cloud.

Your 16GB laptop becomes very slow while `qwen3.5:9b` is loaded next to Docker and a browser with 40 tabs. What are your two options?

Pull the smaller 4B model, or close what you don’t need. A loaded model occupies RAM, and Ollama unloads an idle one after five minutes by default, so you can also run ollama stop on the model when you are done with it.

`ollama serve` fails with "address already in use". What do you check?

Whether the Ollama app is already running in the background. Run curl http://localhost:11434/api/tags: if you get a JSON reply, Ollama is up and you can skip ollama serve.

Your script sends meeting notes full of quotation marks to the local model, and the request breaks. What fixes it?

Build the request with jq, as the local_llm function does. It escapes quotes, line breaks and backslashes for you, so the notes can’t break the JSON.

Part 4 of 4 Keep

Keep this

Send the mechanical jobs local and keep the judgement calls on the cloud

Run a local model for the jobs where a clear prompt gets an answer that is plainly right or wrong, and keep the cloud for anything that needs thinking. You can set up Ollama with a few commands.

• Test a model on a handful of items you have already done yourself before you hand it a batch.
• Pick the smallest model that fits your RAM, and look for newer ones in the Ollama library now and then.
• Keep security reviews, tool chains and writing in your own voice on the cloud.
• Use the draft-first pattern for boilerplate: the local model writes the first draft, and a cloud model applies judgment and polish.

After a month of routing mechanical tasks locally, check your cloud API usage. The tasks still on the cloud bill are the ones that need the bigger model.

In the Kit Library, members can get the shell function, a rules file that tells your assistant which jobs go local, and the RAM table.

Print this card

Read next

• How to Audit Your AI Usage Patterns: find which of your tasks are mechanical enough to go local.
• Context Windows Explained: why a long prompt costs more and how much a model can hold at once.