_Kindi
Today's software doesn't let you run really smart AI models on low-cost hardware. That's where Kindi comes in. Kindi 0.2.0 runs modern 2B to 27B language models on NVIDIA cards from a 1 GB Quadro K420 to an RTX 3080.
Install Kindi 0.2.0 Get it on GitHub Read the manualtok/s write; llama.cpp 1.65
tok/s write; llama.cpp 1.69
tokens of context
InstallWhy Kindi
Kindi started as an experiment: how far can you push an old graphics card? The first card was a 2014 NVIDIA Quadro K420 with 1 GB of memory, which you can buy used for about US$15.
Kindi 0.2.0 has grown from that experiment into a custom engine for a range of hardware. It runs four Ternary-Bonsai-2 models from Prism ML, with 2B, 4B, 9B and 27B parameters. It runs them on the Quadro K420 (1 GB and 2 GB), the Quadro K1200, two K1200 cards that tensor parallelism joins into one 8 GB card, the GeForce GTX 1080 and the GeForce RTX 3080. In many cases, our tests show that Kindi is, to our knowledge, the fastest publicly available inference stack on these cards.
vLLM, another widely used inference server, needs compute capability 7.5 or higher (vLLM installation guide), so it does not support the K420 (3.0), the K1200 (5.0) or the GTX 1080 (6.1).
Kindi has an OpenAI-compatible server for chat programs, scripts and coding agents. Its default output is token-for-token identical to the output of llama.cpp for the same model file. After you download the model once, it needs no internet connection, no account and no subscription.
Cards
Each card here runs Kindi 0.2.0 in our tests. The drawings are to one scale.
NVIDIA Quadro K420
- Memory
- 1 GB DDR3 (a 2 GB model also exists)
- Power
- 41 W, all from the PCIe slot
- Size
- Low profile, one slot
- Models
- 2B on the 1 GB card; 2B or 4B on the 2 GB card
- Speed
- The 2B writes 20.8 tok/s at 512 tokens
- Good for
- Short text decisions, small agent tasks, a private assistant on an old office computer
NVIDIA Quadro K1200
- Memory
- 4 GB GDDR5
- Power
- 45 W, all from the PCIe slot
- Size
- Low profile, one slot
- Models
- 2B, 4B or 9B
- Speed
- At 512 tokens: 81.0 tok/s with the 2B, 39.2 with the 4B, 23.4 with the 9B
- Good for
- Agent work with the 4B, a 9B model in a small office computer
Two Quadro K1200 cards
- Memory
- 8 GB in total, 4 GB on each card
- Power
- 90 W for the two cards, all from the slots
- Size
- Two low-profile slots
- Models
- 27B, one model on both cards
- Speed
- The 27B writes 13.0 tok/s at a short context
- Good for
- A 27B model in a small office computer
NVIDIA GeForce GTX 1080
- Memory
- 8 GB GDDR5X
- Power
- 180 W, one 8-pin PCIe power cable
- Size
- Full height, two slots
- Models
- 27B with a context of 65,536 tokens
- Speed
- The 27B writes 27.9 tok/s at 512 tokens
- Good for
- The 27B on one card, long documents, programming tasks
NVIDIA GeForce RTX 3080
- Memory
- 10 GB GDDR6X
- Power
- 320 W on the NVIDIA design; ours have a default limit of 380 W
- Size
- Full height, two slots or more
- Models
- 27B with about 85,000 tokens of context with the installer's 3080 settings (up to 103,680 without the drafter), and the 2B as a drafter
- Speed
- The 27B writes 63.2 tok/s at a depth of 1K at 190 W, and 66.4 at 380 W (dev build release-tcspec6)
- Good for
- Fast agent work with the 27B, very long documents
Which models run on which cards?
All four models are Prism ML's Ternary-Bonsai-2 PTQ1_0 files, with weights of three values. A larger model makes fewer mistakes. It also needs more memory, and it writes more slowly on the same card. The installer selects the model for your card.
| Model | File size | Card and context (tokens) | Good for |
|---|---|---|---|
| 2B | 429 MB | Quadro K420 1 GB: 24,576. Quadro K420 2 GB: 65,536. Quadro K1200: 24,576 in our tests, fits up to 131,072 | Short decisions and small agent tasks. The fastest model on every card. |
| 4B | 950 MB | Quadro K420 2 GB: 25,088. Quadro K1200: 24,576 in our tests, fits up to 86,272 | Agent work with more steps |
| 9B | 2.0 GB | Quadro K1200: 24,576 in our tests, fits up to 47,360 | A larger model on one small card |
| 27B | 5.9 GB | Two Quadro K1200 cards: 24,576. GeForce GTX 1080: 65,536. GeForce RTX 3080: about 85,000 with the installer's 3080 settings, up to 103,680 without the drafter | Programming tasks and long documents |
Performance
One rule for each comparison: the same card, the same model file and the same context for both engines. In these tests, Kindi writes faster than llama.cpp on every card. It also reads faster on the K420, the K1200, two K1200 cards and the GTX 1080. On the RTX 3080, the two engines read at about the same speed.
Read speed is the speed at which the engine reads your prompt. Write speed is the speed at which it writes the answer. Context is the number of tokens that the model can see. When only one engine reaches a context, or a run crashed, the chart says so. All speeds are in tokens per second (tok/s), one bar scale for each chart, from zero.
Each card in detail, with read speeds and every context:
Builds: K420, GTX 1080 and RTX 3080 Kindi 0.2.0; K1200 dev 7fb99ff; 2x K1200 dev d62501b (read: 7fb99ff). RTX 3080 at 380 W: dev build release-tcspec6 (2026-10-22); its exact path is the same code as 0.2.0, and its drafter path is from before the 0.2.0 draft-model fix.
Quadro K420 1 GB
Ternary-Bonsai-2-2B · f16 K/V · greedy · Kindi exact head
At 512 tokens, Kindi reads 33.3 tok/s and writes 20.8 tok/s. llama.cpp reads 1.2 tok/s and writes 1.65 tok/s. At 8,192 tokens, the llama.cpp run crashed with a Vulkan driver error. Kindi's largest context on this card is 24,576 tokens.
Table and test conditions
| Context (tokens) | Kindi read | llama.cpp read | Kindi write | llama.cpp write |
|---|---|---|---|---|
| 512 | 33.3 | 1.2 | 20.8 | 1.65 |
| 8,192 | 29.7 | crashed | 18.6 | crashed |
| 24,000 | 24.2 | – | 15.3 | – |
| 32,768 | does not fit | loads | does not fit | loads |
| 65,536 | does not fit | does not load | does not fit | does not load |
The same model file (sha256 36061ad4…) and the same 24,576 token ids for both engines, greedy. The engine writes 32 tokens at each depth. llama.cpp: the prebuilt Vulkan release, flash attention on. Kindi: Kindi 0.2.0, exact head, prompts in chunks of 128 tokens. At 32,768 tokens Kindi does not fit and llama.cpp loads; at 65,536 neither engine loads.
Quadro K1200 4 GB
Ternary-Bonsai-2-2B, 4B and 9B · f16 K/V · greedy · Kindi exact head
At 512 tokens, Kindi writes 81.0 tok/s with the 2B, 39.2 with the 4B and 23.4 with the 9B. llama.cpp writes 20.8, 8.7 and 5.9. Kindi also reads several times faster at each context.
Table and test conditions
| Model, context (tokens) | Kindi read | llama.cpp read | Kindi write | llama.cpp write |
|---|---|---|---|---|
| 2B, 512 | 194.5 | 26.7 | 81.0 | 20.8 |
| 2B, 8,192 | 178.7 | 28.1 | 73.3 | 21.9 |
| 2B, 24,000 | 152.3 | 26.5 | 58.8 | 18.6 |
| 4B, 512 | 77.4 | 11.1 | 39.2 | 8.7 |
| 4B, 8,192 | 70.2 | 10.8 | 34.3 | 8.1 |
| 4B, 24,000 | 58.9 | 10.2 | 26.4 | 6.9 |
| 9B, 512 | 41.7 | 5.8 | 23.4 | 5.9 |
| 9B, 8,192 | 39.2 | 5.7 | 21.5 | 5.6 |
| 9B, 24,000 | 35.6 | not finished | 18.1 | not finished |
One Quadro K1200 in a PCIe x16 slot. The same model files and token ids for both engines, greedy. llama.cpp: Prism fork, Vulkan build, flash attention on, a one-hour limit for each model; the 9B run read 8,192 tokens of the 24,000-token prompt and then its limit ran out. Kindi: development build 7fb99ff, exact head, prompts in chunks. 2026-10-22. Largest context for Kindi on this card: 131,072 (2B), 86,272 (4B), 47,360 (9B). llama.cpp loads all three at 65,536, where the Kindi 9B does not fit.
Two Quadro K1200 cards
Ternary-Bonsai-2-27B · q8_0 K/V · greedy · Kindi tensor parallel, llama.cpp layer split
With the same model file on the same two cards, Kindi writes 13.0 tok/s and llama.cpp writes 1.69 tok/s: 7.7 times faster. With the same 512-token prompt, Kindi reads 21.0 tok/s and llama.cpp reads 1.63 tok/s: 12.9 times faster.
Table and test conditions
| Measure | Kindi read | llama.cpp read | Kindi write | llama.cpp write |
|---|---|---|---|---|
| Read 512 / write short | 21.0 | 1.63 | 13.0 | 1.69 |
Two Quadro K1200 4 GB cards in one computer, the same model file (sha256 53107f53…). llama.cpp: Prism fork, Vulkan build, flash attention on, its default layer split; with an even split it runs out of memory. Kindi: the 0.2.0 tensor-parallel engine, exact head, context 24,576. Write 2026-10-21; read 2026-10-22, Kindi build 7fb99ff, best of 3.
GeForce GTX 1080 8 GB
Ternary-Bonsai-2-27B · q8_0 K/V · greedy · Kindi exact head
At 512 tokens, Kindi reads 254.0 tok/s and llama.cpp reads 98.1. Kindi writes 27.9 tok/s and llama.cpp writes 19.4. At 64,000 tokens, llama.cpp runs out of memory, and Kindi writes 15.4 tok/s.
Table and test conditions
| Context (tokens) | Kindi read | llama.cpp read | Kindi write | llama.cpp write |
|---|---|---|---|---|
| 512 | 254.0 | 98.1 | 27.9 | 19.4 |
| 8,192 | 227.1 | 96.2 | 25.7 | 18.1 |
| 32,768 | 169.1 | 84.9 | 19.7 | 14.3 |
| 49,152 | 144.6 | 78.6 | 17.2 | 12.6 |
| 64,000 | 127.8 | out of memory | 15.4 | out of memory |
One card for both engines, the same model file (sha256 53107f53…) and the same token ids, greedy. llama.cpp: Prism fork, CUDA build for sm_61, flash attention on, context 49,280; at 65,600 it does not load. Kindi: Kindi 0.2.0, 5-trit weights, int8 prompt path, exact head, context 65,536.
GeForce RTX 3080 10 GB
Ternary-Bonsai-2-27B · q8_0 K/V · greedy · Kindi exact head, Tensor Cores
At a 190 W power cap, Kindi writes 63.2 tok/s at a depth of 1K and llama.cpp writes 52.2. The difference grows with the depth: at 90K, Kindi writes 44.0 and llama.cpp 23.5. The read speeds are close: at 190 W, Kindi reads faster up to 12,000 tokens, and at 22,000 tokens both engines read 855 tok/s. With the 2B as a drafter, Kindi writes 99.9 tok/s on agent prompts at 190 W, and 108 tok/s at 380 W (dev build release-tcspec6 of 2026-10-22, from before the 0.2.0 draft-model fix). llama.cpp cannot load the 2B as a draft model for the 27B.
Table and test conditions
| Prompt / depth (tokens) | Kindi read | llama.cpp read | Kindi write | llama.cpp write |
|---|---|---|---|---|
| Read 512 / write 1K, 190 W | 1,093 | 936 | 63.2 | 52.2 |
| Read 3,000 / write 32K, 190 W | 1,029 | 995 | 56.3 | 41.0 |
| Read 12,000 / write 64K, 190 W | 944 | 926 | 49.8 | 28.0 |
| Read 22,000 / write 90K, 190 W | 855 | 855 | 44.0 | 23.5 |
| Read 512 / write 1K, 380 W | 1,361 | 1,217 | 66.4 | 56.5 |
| Read 3,000 / write 32K, 380 W | 1,282 | 1,265 | 60.1 | 47.0 |
| Read 12,000 / write 64K, 380 W | 1,171 | 1,170 | 54.7 | 37.9 |
| Read 22,000 / write 90K, 380 W | 1,050 | 1,074 | 51.2 | 32.9 |
One RTX 3080 for both engines, the same model file (sha256 53107f53…), flash attention on. Read: a prompt of N tokens from an empty cache. Write: 32 greedy tokens after a prompt of D tokens. llama.cpp: Prism fork at commit 1180918, CUDA 12.4 build for sm_86. Kindi 0.2.0: exact head, Tensor Core engine. The 190 W rows were measured 2026-10-22 on the v0.2.0 engine, in the same session as llama.cpp; the 380 W rows: dev build release-tcspec6 (2026-10-22), whose exact path is the same code as 0.2.0; its 2B drafter bar is from before the 0.2.0 draft-model fix. Largest context in these runs: Kindi 103,680, llama.cpp 90,000. Agent prompts: 30 greedy requests of 1.1K to 10.5K tokens with 128 tokens out; at 190 W, Kindi's greedy output with the drafter was identical to its exact output in all 30. 2026-10-22.
Which model
Which model for which job? We gave each model the same 65 agent tasks of the Kindi benchmark: 24 terminal tasks, 32 programming and repair tasks, and 9 chains of small prompts. Each step up the model ladder passes more tasks.
Short, clear jobs on a 1 GB card
Short decisions about a text, and terminal tasks with one action and one clear output. Let your program do the exact steps, such as counting and sorting.
Agent work on a 2 GB card
Terminal work with several steps on a 2 GB K420 or a K1200. It passed 48 of the 65 tasks. Give it more time for each task than the 2B.
The largest model on one small card
Programming tasks on one 4 GB card. It passed 26 of the 32 programming and repair tasks.
The most capable model
Programming, long documents and jobs with many steps, on 8 GB of card memory. On the GTX 1080 it passed 63 of the 65 tasks.
| Model | Card | Terminal | Programming and repair | Chains | Write tok/s |
|---|---|---|---|---|---|
| 2B | Quadro K420 1 GB | 13 / 24 | 7 / 32 | 1 / 9 | 20.8 at 512 tokens |
| 4B | Quadro K420 2 GB | 22 / 24 | 23 / 32 | 3 / 9 | 9.8 at position 100 |
| 9B | Quadro K1200, one server on each of two cards | 22 / 24 * | 26 / 32 | – | 23.4 at 512 tokens, one K1200 |
| 27B | GeForce GTX 1080 | 24 / 24 | 32 / 32 | 7 / 9 | 27.9 at 512 tokens |
kbench with the pi agent, thinking on, 4,096 output tokens, one run of each task, exact head. * A composite: the first run used the fast head, and the 6 failed tasks ran again with the exact head. The 2B terminal tasks ran on Kindi 0.1.0. Check each result yourself: a model can say that a job is complete when it is not. One Kindi server answers one request at a time.
Install
One command. The installer finds your cards, selects the model and the settings, downloads and checks the model, converts it, checks that the output matches llama.cpp token for token, and starts the server on 127.0.0.1:8093.
# Linux x86-64 with an NVIDIA card from the list above, or no card (the CPU engine)
curl -fsSL https://amirhusain.com/kindi/install.sh | bashDry run first
# see each step and each command first; it changes nothing
curl -fsSL https://amirhusain.com/kindi/install.sh | bash -s -- --dry-runCheck the script
# or download it, check its sha256, read it, then run it curl -fsSLO https://amirhusain.com/kindi/install.sh echo "e20573338e7d8048387265ef5958cc70ad3c23862e709f6127e32bc424ad22fb install.sh" | sha256sum -c - install.sh: OK bash install.sh
What you need
A Linux computer (x86-64) with the NVIDIA driver (the 470 driver for Kepler cards such as the K420) and Docker. If
Docker or the NVIDIA Container Toolkit is missing, the installer asks you before it installs it. Free disk space for the model: from 429 MB
for the 2B to 5.9 GB for the 27B. The server listens on 127.0.0.1 only, and the installer asks for no keys. To remove Kindi, type
bash install.sh --uninstall.
The model for your card
| Your card | Model | Context (tokens) |
|---|---|---|
| No NVIDIA card | 2B on the CPU | 24,576 |
| Quadro K420 (1 GB or 2 GB) | 2B | 24,576; 65,536 on the 2 GB card |
| One Quadro K1200 | 9B | fitted to the free memory |
| Two Quadro K1200 cards | 27B, tensor parallel | 24,576 |
| GeForce GTX 1080 | 27B | 65,536 |
| GeForce RTX 3080 | 27B, Tensor Cores and the 2B drafter | fitted to the free memory |
To select a different model, add --model 2b, 4b, 9b or 27b.
Use it
# any OpenAI client; use the model id that the installer shows from openai import OpenAI client = OpenAI(base_url="http://localhost:8093/v1", api_key="none") r = client.chat.completions.create(model="bonsai2-2b", messages=[{"role": "user", "content": "Write a bash one-liner that counts .txt files"}]) print(r.choices[0].message.content) ls *.txt | wc -l
Release and source
The v0.2.0 release has the source archive and the release notes. The repository has the documentation for each card, Docker and native installs, the pi agent setup and troubleshooting.
Kindi is free software under the AGPL-3.0. Individuals, researchers and anyone porting it to other GPUs can use, change and share it at no cost. Commercial licenses are available for closed products and hosted services: get in touch.
Field manual
How To Get Work Done with Kindi and a $15 GPU
The second edition, for Kindi 0.2.0. Which card to use for which job, the full performance matrix, two cards as one engine, power caps and the installer. Then how to work with Kindi and the pi agent: which jobs work, which jobs to avoid, how to break a big job into steps, and which model to use for which job. Every speed in it comes from a measurement.
The first edition, for Kindi 0.1.0 on the Quadro K420, is still available.
Live on a K420
Recorded live with Kindi 0.1.0: the pi coding agent, with a Kindi colour theme, driving kindi serve on one Quadro K420 at default settings, each task in a fresh session. It runs nvidia-smi, writes and runs a Python script and checks disk space. Token streaming is real time; pauses longer than 2.5 seconds are shortened. Watch the full, untrimmed recording (123 seconds).
What you get
OpenAI-compatible
kindi serve speaks /v1/chat/completions with streaming and tool calls. Chat apps, scripts and coding agents work unchanged, and nothing leaves the machine.
Exact by default
Greedy output matches llama.cpp token for token on the same model file. The installer runs kindi verify on your card before it starts the server.
Built for agents
Checkpoints, parked conversations and saved openings mean an agent's long system prompt is read once, not on every turn, even across restarts.
Two cards as one
Tensor parallelism cuts each matrix in two and puts one half on each card, so both cards work all the time. Two low-profile K1200s hold the 27B.
Speculative decoding
On the RTX 3080, the 2B drafts tokens for the 27B, and the 27B checks them in one pass. With greedy decoding, the output is the same as the output without the drafter: on 30 greedy agent prompts at 190 W, all 30 were identical.
Small and self-contained
One Python package and one CUDA source file, a Docker image for the CUDA 10.2 stack that Kepler needs, and a CPU engine for computers with no NVIDIA card.
How it works
No unpacking
Ternary weights are never unpacked: four weights are one byte that indexes a table of precomputed partial sums, one shared-memory read per four weights.
No collisions
Every lane of a warp owns its tables, laid out in its own memory bank, so no two threads ever collide, and one byte-permute instruction turns a weight byte into its table address.
Fractal tables
Each table is built level by level as three shifted copies of the level below: the ternary construction of the Cantor set, 27 additions with no division.
Compact 27B
A 5-trit format stores the 27B weights so that the model and a 65,536-token context fit in 8 GB, and prompts are read in chunks, with int8 arithmetic on sm_61 and later cards.
The story
Why an engine for a ten-year-old graphics card? Because careful engineering lets a low-cost card do real work. I talk about Kindi, ternary models and the Quadro K420 on my channel.
Watch the launch video Deep Tech with Amir on YouTube All software