Kortex: Streaming 70B Models on Consumer GPUs

Kortex, a new Rust-based LLM inference engine, reframes large model deployment as a streaming problem. By doing so, it runs 70B-parameter models on consumer GPUs with just 20GB of VRAM, outperforming established solutions for out-of-core tasks.

The key insight: token generation is memory-bandwidth-bound, not compute-bound. Kortex's architecture is built entirely around this fact:

Instead of partially offloading to CPU when VRAM is exceeded, Kortex keeps all computation on the GPU. It streams model weights through a pipeline from NVMe to RAM to VRAM, overlapping data transfer with computation. This approach pays off dramatically for large models:

| Model         | Weights | Kortex    | llama.cpp |
|---------------|---------|-----------|-----------| 
| Llama-3.3-70B | 42.5 GB | 1.95 tok/s| 0.21 tok/s|

The 9x speed advantage isn't incremental; it's a qualitative shift in what's possible on consumer hardware. While llama.cpp hits a hard ceiling due to DRAM bandwidth when offloading, Kortex's all-GPU approach scales to much larger models.

However, it's not a universal win. For models fitting entirely in VRAM, llama.cpp still holds an edge:

| Model        | Weights | Kortex   | llama.cpp |
|--------------|---------|-----------|-----------| 
| Llama-3.1-8B | 4.9 GB  | 89 tok/s  | 109 tok/s |

This reflects llama.cpp's maturity and optimization for in-VRAM scenarios. Kortex's niche is specifically pushing beyond VRAM limits.

Key components enabling this performance:

1. Residency planner: Decides where each tensor lives (VRAM, RAM, or NVMe).
2. Streaming pipeline: Moves layers through VRAM slots, overlapping I/O and compute.
3. Quantized end-to-end: Dequantizes weights only in GPU registers.
4. Speculative decoding: Boosts effective tokens per second.

Notably, Kortex isn't a wrapper. Its GGUF parser, tokenizer, quantized kernels, and GPU backend are all original code. This from-scratch approach allows for deep optimization but also limits scope: it's Windows-only for now and supports a narrower range of models and quantizations.

The significance extends beyond current capabilities. It demonstrates a viable path for running state-of-the-art language models on widely available hardware. This could open up access to advanced AI capabilities, enabling more developers and researchers to work with advanced models without data center-grade GPUs.

However, Kortex remains an early-stage project with limitations. It lacks multi-turn chat capabilities, an HTTP server for easy integration, and support for some popular quantization formats. Its Windows-only status also restricts immediate applicability.

For developers and researchers working with large language models, Kortex is a compelling option when dealing with models exceeding local GPU memory. Its performance on these out-of-core tasks makes it a valuable tool, complementing rather than replacing solutions like llama.cpp for in-VRAM scenarios.

As the project evolves, addressing current limitations and expanding platform support could position Kortex as a key enabler for local deployment of large AI models, potentially reshaping the landscape of AI accessibility and experimentation.