Skip to content

All pages

Running an LLM on your own computer

5 tools, Last updated 2026-09-16

With local LLMs the first question is not which tool but which model fits in your memory. The same model name can be several times larger depending on quantization, and the KV cache eats more memory as the conversation grows. The tools below split by how they handle that: one measures your hardware and picks models for it, one streams weights from the SSD, and some are cut down around a single model.

Korean is decided by the model, not the tool. A server with a Korean UI is no help if the model writes poor Korean. So the Korean section below looks at what each tool's default target model says about Korean on its official model card.

How this was checked. Written on 2026-09-14 from each tool's GitHub README and the official model cards (Google, DeepSeek, Qwen, Hugging Face). Entries we installed and ran are marked "Installed and tried"; the rest are "From documentation". Tokens-per-second figures are only those the README states together with the machine.

At a glance

InstallOmlxFrom documentationMagnitudeFrom documentationTurbo FieldfareFrom documentationDs4From documentationAirllmFrom documentation
Install.dmg or brew, macOS 15+One npm lineXcode 26 source build + 15GB modelmake build, project-specific GGUF downloadpip install airllm
KoreanAdmin UI available in KoreanNot mentioned in the READMEGemma 4 card does not name KoreanDeepSeek V4 card lists no languagesNo mention of Korean in the README
Key constraintMLX models only, Apple Silicon onlyBackend undisclosed; Windows via WSLmacOS 26; Gemma 4 26B-A4B only96GB+ Mac as the Metal baselineSlow: streams one layer at a time from disk

Omlx

Apple Silicon LLM inference server with continuous batching, run from the macOS menu bar 21,850From documentation

Pick this when. When one Apple Silicon Mac should serve several apps or a team as an API server, managed from the menu bar instead of a terminal.

Installation

  1. macOS 15 or newer on Apple Silicon (M1 onwards). No Intel Macs, Windows or Linux.
  2. Download the .dmg and drag it into Applications; it updates itself in-app. With Homebrew: brew tap jundot/omlx, then brew install jundot/omlx/omlx.
  3. From source, pip install -e . on Python 3.11 to 3.13; add ".[mcp]" for MCP support.
  4. Only MLX-format models. The admin dashboard searches Hugging Face for MLX models and downloads them directly.

Korean-language usability

The README says the admin UI supports eight languages including Korean. Korean output quality depends on the model you load. The Qwen3.5 family used in its examples lists Korean among its main languages in public material, but whichever model you pick, check the language section of its card first.

Real constraints

  • MLX only; GGUF files will not load.
  • The default memory limit is system RAM minus 8GB; past that it evicts the least recently used model first.
  • Custom kernels for newer models such as GLM-5.2 and MiniMax M3 are not built by a plain pip install and need the full Xcode.
  • Speaks both the OpenAI and Anthropic API formats. Apache 2.0.

Checked on 2026-09-14, Checked: From documentation. Written from the official docs and README. Not installed and tried by us.

Magnitude

Open source inference server that picks local models to fit your hardware 4,619From documentation

Pick this when. When you do not know which model to pick. It measures your machine, shortlists what will run, and plugs into the coding agent you already use.

Installation

  1. npm i -g @magnitudedev/cli is the whole install. The README states no Node.js version requirement.
  2. Runs on macOS, Linux and Windows (via WSL), targeting Apple Silicon, NVIDIA and AMD GPUs and CPU-only machines.
  3. After install it profiles your chip, memory and bandwidth, shows which models will run with an estimated tokens per second, then downloads and serves the one you pick.

Korean-language usability

The README says nothing about languages. It ranks models by speed, accuracy and memory, so for Korean you have to pick a model from its shortlist that lists Korean as supported (Qwen or Gemma families, for example). We could not confirm a Korean UI either.

Real constraints

  • The README does not say which inference backend (llama.cpp, MLX or other) it uses.
  • Windows goes through WSL.
  • Plugs into Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi and Cline, with a built-in harness if you use none of them. Apache 2.0. No token bill, API key or rate limit.

Checked on 2026-09-14, Checked: From documentation. Written from the official docs and README. Not installed and tried by us.

Turbo Fieldfare

Gemma 4 26B-A4B inference in about 2 GB of RAM on an M-series MacBook. 6,766From documentation

Pick this when. When you want to try a 26B-class model on a small-memory M-series Mac such as an 8GB MacBook Air.

Installation

  1. Needs macOS 26 (Metal 4) on Apple Silicon. The validated target is an 8GB M2 MacBook Air.
  2. Source build only: swift build -c release with Xcode 26 and Swift 6.2 or newer, then run .build/release/TurboFieldfareMac.
  3. The app's Download button fetches the model and repacks it into .gturbo format: about 15GB transferred, about 14.3GB installed, plus 1.1GB for the optional image pack.
  4. The Mac app, the CLI and the OpenAI-compatible Chat Completions server all share the same model folder.

Korean-language usability

The model is fixed: Gemma 4 26B-A4B. Google's model card says it was pretrained on more than 140 languages but does not name Korean individually. The README says nothing about languages.

Real constraints

  • 8GB of RAM minimum; the "about 2GB" figure covers the weights and a 4K KV cache. Keep more than 15GB of disk free.
  • Speeds stated in the README: 5.1 to 6.3 tok/s on an 8GB M2 MacBook Air, 31 to 35 tok/s on a 24GB M5 Pro.
  • Text in, images optional on M2 or newer; no audio or video. M1 is text only.
  • The server is loopback only with no authentication or TLS. Apache 2.0; the model weights follow the Gemma terms.

Checked on 2026-09-14, Checked: From documentation. Written from the official docs and README. Not installed and tried by us.

Ds4

Local inference engine built narrowly for DeepSeek V4. 22,494From documentation

Pick this when. When you own a big machine, a 96GB-plus Mac or a DGX Spark, and want DeepSeek V4 to run as fast as it can.

Installation

  1. From the repo root, pick one of make (Metal), make cuda-spark (DGX Spark), make strix-halo (AMD Strix Halo) or make cuda-generic (multi-GPU CUDA).
  2. Models come from the project's own GGUF files, e.g. ./download_model.sh ds4f-q2. It is not a general GGUF runner, so GGUF files from elsewhere will not work.
  3. ds4-server serves at http://127.0.0.1:8000, the CLI is an interactive chat, and ds4-agent is a coding agent with tool calls.

Korean-language usability

The official DeepSeek V4 Flash card lists no supported languages and only publishes Chinese and English benchmarks (CMMLU, C-Eval and others). GLM 5.x and Qwen3.8 are also supported, but this tool's README says nothing about languages. Korean quality should be treated as unconfirmed.

Real constraints

  • The Metal baseline is a Mac with 96GB or more; smaller machines fall back to SSD streaming. Even 128GB machines need SSD streaming for the full GLM 5.x models.
  • DeepSeek V4 Flash has 284B total and 13B active parameters. Qwen3.8 Flash Next at Q2 is 41.73GiB of main weights plus 95.37GiB of n-grams.
  • The README calls itself beta quality, and sessions containing images cannot be saved yet.
  • MIT licensed; the DeepSeek V4 Flash model is MIT too.

Checked on 2026-09-14, Checked: From documentation. Written from the official docs and README. Not installed and tried by us.

Airllm

70B-parameter inference on a single 4GB GPU. 34,488From documentation

Pick this when. When you have only 4GB of GPU memory and still need to run a 70B-class model once. An experiment tool where running at all matters more than speed.

Installation

  1. pip install airllm. For 4-bit and 8-bit compression, also pip install -U bitsandbytes.
  2. On a Mac you need both mlx and torch, and only Apple Silicon is supported.
  3. Some models need a specific transformers build (Qwen3.8-Flash-Next needs the latest from git, for example). Gated models need a Hugging Face token.

Korean-language usability

It supports a wide range of model families: Llama, Qwen, DeepSeek, Mistral, Gemma, ChatGLM, Baichuan, Yi and more. The README names several Chinese models but never mentions Korean. For Korean, check the model card of the model you intend to use before choosing. It is a library, so there is no UI.

Real constraints

  • It keeps only one layer on the GPU at a time and streams the rest from disk, so it is slow. The README only says compression gives a 3x speedup and states no absolute speed.
  • Models are split into per-layer files, which takes a lot of disk; running out stops it with a MetadataIncompleteBuffer error.
  • Some models have no padding token and need it set by hand, and Kimi K3 requires CUDA 12.
  • CPU inference gets a single line of mention and no detail. Apache-2.0.

Checked on 2026-09-14, Checked: From documentation. Written from the official docs and README. Not installed and tried by us.