Español Login Register

How to Run a Local LLM: Ollama, llama.cpp, LM Studio, and vLLM Compared

DaDesktop Team ·

There are multiple approaches to executing LLMs locally. While some tools are optimized for rapid onboarding, others prioritize granular control or are engineered to handle high-volume user traffic. Your selection will depend on whether you require a straightforward local chat interface, a highly configurable inference engine, or a robust production API.

Ollama

Ollama stands out as one of the most accessible methods for initiating local model execution. The process is streamlined: install the software, download a model, and execute it via the command line. Additionally, it offers a local API to support applications and other integrations.

Pros:

  • Straightforward installation and model management
  • Intuitive command-line workflow
  • OpenAI-compatible API support
  • GPU acceleration support for NVIDIA, AMD, Apple Silicon, and Vulkan
  • Modelfiles enable customization of models and parameters
  • Capable of handling concurrent requests, provided sufficient memory is available

Cons:

  • Offers less low-level control compared to llama.cpp
  • Model management is tightly integrated with the Ollama ecosystem
  • Not the primary choice when maximum serving throughput or distributed inference is required

Difficulty: Low. Ideal for those seeking to run a model quickly without navigating complex inference configurations.

llama.cpp

llama.cpp is a lightweight C/C++ inference engine designed for efficient model execution across diverse hardware platforms. It utilizes GGUF models and provides detailed control over model loading and runtime behavior.

Pros:

  • Granular control over context, GPU offloading, batching, threads, quantization, and other inference parameters
  • Extensive hardware support, including CUDA, HIP, Metal, Vulkan, and SYCL
  • Supports a wide range of quantization levels, from low-bit formats up to 8-bit
  • Ability to distribute models across multiple GPUs
  • Supports hybrid CPU/GPU execution when models exceed available VRAM
  • Includes llama-server for an OpenAI-compatible API

Cons:

  • Requires more configuration than Ollama or LM Studio
  • GGUF models typically need to be downloaded and managed separately
  • Many advanced settings demand a solid understanding of inference parameters

Difficulty: Medium. A suitable choice for those who wish to precisely manage model execution or experiment with performance and quantization strategies.

LM Studio

LM Studio is a desktop application designed for downloading, configuring, and executing local LLMs. It features a graphical interface that simplifies model discovery and management of settings such as GPU offloading and context size.

Pros:

  • User-friendly graphical interface
  • Model search and download capabilities via Hugging Face
  • Presents model and resource details prior to loading
  • OpenAI-compatible API server
  • Supports headless execution through the llmster server
  • Supports GGUF models via llama.cpp and MLX models on Apple Silicon

Cons:

  • Offers less low-level control than direct use of llama.cpp
  • The desktop format may be less ideal for certain server deployments
  • Not primarily optimized for large-scale multi-user serving

Difficulty: Low. Perfect for those who prefer experimenting with local models without extensive command-line interaction.

vLLM

vLLM is engineered for serving LLMs to applications and multiple users concurrently. Its key strength lies in efficient high-concurrency serving, leveraging techniques such as PagedAttention, continuous batching, prefix caching, and distributed inference.

Pros:

  • High throughput for handling multiple concurrent requests
  • Continuous batching and optimized KV-cache management
  • OpenAI-compatible API server
  • Direct compatibility with numerous Hugging Face models
  • Support for quantization formats including FP8, INT4, GPTQ, AWQ, and GGUF
  • Support for tensor, pipeline, expert, and other parallelism strategies
  • Designed for production-grade inference and serving

Cons:

  • More complex setup and configuration process
  • Primarily targeted at Linux environments
  • Often overkill for a single user running one model interactively
  • Requires verification of hardware and model compatibility prior to deployment

Difficulty: High. Best suited for deploying inference services rather than running models on a personal computer.

Which one should you pick?

  • For easy model execution: Ollama or LM Studio. Choose Ollama for command-line simplicity and a basic API, or LM Studio for a graphical interface.
  • For inference control: llama.cpp. It provides direct management over model loading, quantization, context, GPU offloading, and other settings.
  • For a local API: Ollama, llama.cpp, or LM Studio. All three offer OpenAI-compatible APIs.
  • For multi-user serving: vLLM. Its continuous batching and distributed inference features are tailored for this scenario.
  • For quantization experimentation: llama.cpp or LM Studio.

Run it on DaDesktop

If local GPU hardware is insufficient, these tools can be executed on a DaDesktop cloud desktop. Select a GPU with adequate VRAM for your chosen model, launch the desktop environment, and install your preferred inference software.

Ollama and LM Studio are advantageous for establishing simple local environments. llama.cpp offers greater control over hardware and inference settings. vLLM is a viable option when exposing a model as a high-throughput API is necessary.

View available GPUs to compare VRAM and other specifications.