How to Use Ollama to Run AI Models Locally: A Beginner’s Setup Guide

Why Running AI Models Locally Actually Makes Sense

Running AI models locally used to mean expensive GPU clusters and a PhD in systems engineering. That’s no longer true. Tools like Ollama have made it genuinely straightforward to run open-weight models on your own hardware — no cloud subscription, no data leaving your machine, no rate limits.

If you’ve been curious about local AI inference but weren’t sure where to start, this guide walks through the full setup. You’ll have Ollama running and a model responding to prompts in under 15 minutes.

The reasons people choose to run AI models locally vary. Some are dealing with sensitive data and can’t send it to a third-party API. Others want to cut costs on high-volume workflows. Some just want to experiment freely without worrying about token bills. Whatever your reason, Ollama is one of the cleanest ways to get there.

---

What Ollama Is (and What It Isn’t)

Ollama is an open-source tool that lets you download, manage, and run large language models (LLMs) on your local machine. It handles the complexity of running quantized model files — packaging everything from model weights to runtime into a simple command-line interface.

Think of it like Docker, but for AI models. You pull a model by name, run it, and interact with it via a local API or directly in the terminal.

What Ollama is not:

• A cloud AI service
• A fine-tuning platform
• A visual interface (though third-party UIs work with it)
• A training tool

It’s purely an inference engine for running pre-trained models. The models it supports are open-weight models — meaning the weights are publicly available — including Llama 3, Gemma, Mistral, Phi, Qwen, DeepSeek, and dozens more.

---

What You Need Before You Start

Before installing Ollama, check that your setup meets the basic requirements.

Operating System

Ollama supports:

• macOS 11 Big Sur or later (works on both Intel and Apple Silicon)
• Windows 10 or later (64-bit)
• Linux (most modern distributions, including Ubuntu 20.04+)

Hardware

This is where things vary most. The short version: more RAM and a decent GPU help, but they’re not always required.

• Minimum: 8GB RAM (for smaller models like Phi-3 Mini or Gemma 2B)
• Recommended: 16GB RAM (for mid-size models like Llama 3.1 8B)
• For larger models: 32GB+ RAM or a GPU with 16GB+ VRAM

If you have an Apple Silicon Mac (M1, M2, M3, or M4 chip), you’re in a good position. Apple’s unified memory architecture means the GPU and CPU share the same RAM pool, making local inference noticeably faster than on comparable Intel hardware.

On Windows and Linux, Ollama will automatically use NVIDIA or AMD GPUs if available. If no GPU is detected, it falls back to CPU inference — which works, just slower.

Disk Space

Models range from about 2GB (small quantized models) to 70GB+ (large models). Make sure you have enough free disk space before pulling a model. A practical starting point is 10–15GB free for a mid-sized model.

---

How to Install Ollama

Installation is quick on all three platforms.

macOS

1. Go to ollama.com and download the macOS app.
2. Open the downloaded .zip file and drag the Ollama app to your Applications folder.
3. Launch the app — you’ll see an Ollama icon in your menu bar.
4. Open Terminal and verify the install by running:

ollama --version

Windows

1. Download the Windows installer from ollama.com.
2. Run the .exe installer — it sets up Ollama as a background service automatically.
3. Open Command Prompt or PowerShell and verify with:

ollama --version

Linux

Ollama provides a one-line install script:

curl -fsSL https://ollama.com/install.sh | sh

This installs the Ollama binary and sets it up as a systemd service so it runs in the background. After installation, verify it’s running:

ollama --version
systemctl status ollama

Once installed on any platform, the Ollama server runs locally and listens on http://localhost:11434 by default.

---

Downloading and Running Your First Model

With Ollama installed, you’re ready to pull a model and run it.

Pull a Model

Use the ollama pull command followed by the model name:

ollama pull llama3.2

This downloads the model file to your local machine. The default version is usually a 4-bit quantized model, which balances quality and file size. For llama3.2, that’s around 2GB.

You’ll see a progress bar as it downloads. Once complete, the model is cached locally — you won’t need to download it again.

Run an Interactive Chat Session

Start a conversation directly in the terminal:

ollama run llama3.2

You’ll see a prompt (>>>) where you can type your message and get a response. To exit, type /bye.

That’s it. You’re now running a local LLM on your own hardware.

Run a One-Off Prompt

If you don’t want an interactive session, pipe input directly:

echo "Explain quantum entanglement in two sentences." | ollama run llama3.2

This outputs the response and exits — useful for scripting and automation.

---

Exploring the Ollama Model Library

Ollama’s model library covers a wide range of open-weight models. Here are some worth knowing about:

Llama 3.2 and 3.1 (Meta)

Meta’s Llama series remains one of the most capable open-weight families. Llama 3.2 includes 1B and 3B versions optimized for edge devices, while Llama 3.1 comes in 8B, 70B, and 405B sizes. The 8B model is a good everyday choice — fast enough on most machines, strong enough for most tasks.

ollama pull llama3.2        # 3B, ~2GB
ollama pull llama3.1:8b     # 8B, ~4.7GB

Gemma 3 (Google)

Google’s Gemma models punch above their weight class for their size. Gemma 3 comes in 1B, 4B, 12B, and 27B variants. The 4B version is a strong choice for general tasks on machines with 8–16GB RAM.

ollama pull gemma3:4b

Mistral and Mixtral (Mistral AI)

Mistral 7B is fast and capable — a reliable workhorse for text tasks. Mixtral 8x7B uses a mixture-of-experts architecture for higher quality, but requires more memory.

ollama pull mistral
ollama pull mixtral

Phi-4 (Microsoft)

Microsoft’s Phi series focuses on efficiency — smaller models trained on high-quality data. Phi-4 Mini is notable for delivering strong reasoning performance at just 3.8B parameters.

ollama pull phi4-mini

DeepSeek-R1

DeepSeek-R1 is worth highlighting for reasoning tasks. It uses chain-of-thought reasoning explicitly and performs well on logic, math, and code tasks.

ollama pull deepseek-r1

Code-Specific Models

If you’re using local AI primarily for coding help:

ollama pull qwen2.5-coder:7b   # Strong code model from Alibaba
ollama pull codellama           # Meta's code-focused Llama variant

You can browse the full library at ollama.com/library, which lists all available models along with their sizes, tags, and parameter counts.

---

Using Ollama Beyond the Command Line

The terminal interface is fine for quick tests, but most real workflows use Ollama’s API or connect it to a frontend.

The Ollama REST API

Once the Ollama server is running, it exposes a local REST API at http://localhost:11434. You can send requests to it from any application that can make HTTP calls.

Generate a completion:

curl http://localhost:11434/api/generate 
  -d '{
    "model": "llama3.2",
    "prompt": "What is the capital of Japan?",
    "stream": false
  }'

Chat endpoint (OpenAI-compatible):

curl http://localhost:11434/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "model": "llama3.2",
    "messages": ({"role": "user", "content": "What is the capital of Japan?"})
  }'

The /v1/ endpoint is OpenAI-compatible, which means any tool or library that supports the OpenAI API can point to Ollama instead — just change the base URL.

Connecting a UI

If you prefer a chat interface, several open-source frontends work directly with Ollama:

• Open WebUI — A feature-rich web interface that connects to Ollama out of the box
• Enchanted — A macOS native app with a clean chat interface
• Jan — A cross-platform desktop app with built-in Ollama support
• Anything LLM — Adds document chat (RAG) on top of local models

These frontends all communicate with Ollama via the local API — no additional configuration needed beyond pointing them at http://localhost:11434.

Using Ollama with Python

The official Python library makes integration simple:

import ollama

response = ollama.chat(
    model='llama3.2',
    messages=({'role': 'user', 'content': 'Summarize this text: ...'})
)
print(response('message')('content'))

Install it with pip install ollama.

Managing Models

Useful commands for day-to-day model management:

ollama list              # See all locally installed models
ollama rm llama3.1:8b   # Delete a model to free disk space
ollama show llama3.2     # View model metadata and system prompt
ollama ps                # See which models are currently loaded

---

Troubleshooting Common Issues

Local AI inference has a few recurring pain points. Here’s how to handle the most common ones.

”Model not found” error

This usually means a typo in the model name, or you’re trying to run a model you haven’t pulled yet. Check your installed models with ollama list first.

Very slow inference

If responses are taking 30+ seconds per token, you’re likely running purely on CPU. Check whether Ollama is detecting your GPU:

ollama ps

If GPU layers show 0, Ollama isn’t using your GPU. On NVIDIA systems, make sure you have the latest CUDA drivers installed. On macOS, GPU acceleration happens automatically with Apple Silicon.

Alternatively, switch to a smaller model — 3B instead of 8B, for example.

Out of memory errors

If your system runs out of RAM or VRAM, the model won’t load. Options:

• Use a smaller quantized version (e.g., llama3.1:8b-instruct-q4_0 instead of the default)
• Try a smaller model entirely
• Close other memory-heavy applications before loading the model

Ollama server not running

If CLI commands fail with a connection error, the background service may have stopped. Restart it:

• macOS: Relaunch the Ollama app from Applications
• Linux: sudo systemctl restart ollama
• Windows: Check that the Ollama service is running in Task Manager or restart it from the system tray

Models not updating

Ollama caches models locally. To get the latest version of a model:

ollama pull llama3.2

Running pull again on an already-installed model checks for updates and downloads newer versions if available.

---

Where MindStudio Fits with Local Models

If you’ve set up Ollama and want to start building actual workflows around your local models, MindStudio is worth knowing about.

MindStudio’s AI Media Workbench explicitly supports local model backends — including Ollama, ComfyUI, and LMStudio — alongside its library of 200+ hosted models. That means you can build automated pipelines that route to your local Ollama instance for privacy-sensitive steps, and to hosted models like Claude or GPT-4o for other tasks.

In practice, this is useful for situations like:

• Running a sensitive document through a local Llama model for summarization, then using a hosted model for the final formatting step
• Building an internal knowledge base tool where all processing stays on your own infrastructure
• Experimenting with different model backends without rewriting your workflow logic

Beyond model routing, MindStudio’s no-code builder lets you connect AI outputs to real business tools — Google Workspace, Slack, Airtable, HubSpot — without writing integration code. So the workflow you prototype locally can connect to the rest of your stack with minimal friction.

You can try MindStudio free at mindstudio.ai. Building a basic workflow typically takes 15–30 minutes, and you don’t need to touch code unless you want to.

For teams already building with local models who want to add automated AI workflows without spinning up infrastructure, it’s a practical bridge.

---

Frequently Asked Questions

Is Ollama free to use?

Yes. Ollama is fully open-source and free to download and use. The models it runs are also free, though licensing varies by model. Most models like Llama 3.2 and Gemma 3 are free for personal and commercial use under their respective licenses. A few models have more restrictive terms — check the model card before using one commercially.

Can I run Ollama without a GPU?

Yes. Ollama runs on CPU if no GPU is available. Performance is slower — a 7B model might generate 3–5 tokens per second on CPU versus 30–60 on a modern GPU — but it works. For lighter tasks or smaller models (1B–3B parameters), CPU-only inference is often fast enough to be practical.

What’s the difference between Ollama and running models in the cloud?

Cloud APIs (OpenAI, Anthropic, Google) run models on remote servers and charge per token. Ollama runs models entirely on your local machine — no data is sent anywhere, there’s no per-token cost, and there are no rate limits. The tradeoff is that cloud models tend to be more capable and require no hardware investment.

Which Ollama model should I start with?

For general use, llama3.2:3b is a good starting point — it’s small enough to run on most machines and capable enough for everyday tasks. If you have 16GB+ RAM and want better quality, llama3.1:8b is a significant step up. For coding specifically, qwen2.5-coder:7b is worth trying.

Can Ollama handle multimodal inputs (images)?

Yes, for models that support it. Llava and BakLLaVA are vision-language models available through Ollama that can accept image inputs alongside text prompts. You can pass image paths via the API or use frontends like Open WebUI that support image uploads.

How do I update models in Ollama?

Run ollama pull again. Ollama will check if there’s a newer version and download it if so. Your existing model version stays until the update completes, so there’s no downtime during the pull.

---

Key Takeaways

• Ollama makes local AI inference accessible on Mac, Windows, and Linux with a simple install and single command to run models.
• System requirements matter — 16GB RAM is a practical starting point for mid-sized models, and Apple Silicon Macs are particularly well-suited for this.
• The model library covers most major open-weight families: Llama, Gemma, Mistral, Phi, DeepSeek, and more.
• Ollama’s OpenAI-compatible API means you can point existing tools and libraries at your local instance with minimal reconfiguration.
• For building workflows around local models — or mixing local and cloud inference — MindStudio’s support for Ollama as a backend is a practical option worth exploring.

If you want to go further than just chatting in a terminal, MindStudio lets you connect your local models to real workflows and business tools without writing infrastructure code. Start free at mindstudio.ai.