Local LLMs, page 3

CPU-first LLM inference server on llama.cpp. Runs useful models on free-tier ARM boxes; rewriting the input made it ~6x faster and more accurate than tuning the engine. MIT, benchmarks and failures included.

114 stars0 this week

Train a tiny Transformer from scratch on your iMessage history, entirely on your Mac.

473 stars+1 this week

MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert cache with LRU hot experts, OpenAI-compatible proxy, GGUF multi-shard.

146 stars+1 this week

A Lightweight LLM Inference Performance Simulator

103 stars0 this week

Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere

1,427 stars+8 this week

🦀 Decoder-only LLM built from scratch in pure Rust using Candle — no Python, no PyTorch. Gated DeltaNet + sparse attention, fine-grained MoE, native video/document understanding, long-horizon tool agents, quantization-aware training. Scales: Tiny (25M) to Large (1.3B).

112 stars0 this week

A minimal interface for AI Companion that runs entirely in your browser.

222 stars+1 this week

llama.cpp optimizations for IBM POWER8: vec_perm non-bijunctive collapse, PSE hardware entropy, DCBT prefetch. Sovereign inference. Part of the Proof of Physical AI stack.

130 stars0 this week

Make Discord your LLM frontend

839 stars+3 this week

Entrpi/ds4, a Blackwell CUDA perf fork of antirez/ds4 on NVIDIA DGX Spark: one-command install, ~3x upstream prefill, ~1.5x decode, DSpark, and full continuous batch support

414 stars+1 this week

Legend of Elya — N64 game with a real 6.36M-parameter ternary transformer on the VR4300 MIPS III CPU. Zelda-style dungeon, AI NPCs, byte-level inference at 1.23 tok/s scalar / 2.19 tok/s on the RSP overlay (measured under ares, never on silicon). Built with libdragon.

142 stars0 this week

Heterogeneous prefill/decode for DeepSeek-V4-Flash: CUDA prefill (DGX Spark, vLLM) -> Metal decode (Mac Studio, oMLX) over plain 10GbE

151 stars0 this week

lcpp is a dart implementation of llama.cpp used by the mobile artificial intelligence distribution (maid)

124 stars0 this week

LLM Inference analyzer for different hardware platforms

129 stars0 this week

Evidence-aware on-prem LLM inference sizing and TCO calculator

109 stars0 this week

LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.

883 stars0 this week

High-speed Large Language Model Serving for Local Deployment

9,874 stars+10 this week

LLMX; Easiest 3rd party Local LLM UI for the web!

297 stars+1 this week

[NeurIPS 2026 Spotlight] Fast, lossless LLM inference via dual-view diffusion decoding.

483 stars0 this week

A portable terminal AI interface

202 stars+1 this week

LLM Agent Framework in ComfyUI includes MCP sever, Omost,GPT-sovits, ChatTTS,GOT-OCR2.0, and FLUX prompt nodes,access to Feishu,discord,and adapts to all llms with similar openai / aisuite interfaces, such as o1,ollama, gemini, grok, qwen, GLM, deepseek, kimi,doubao. Adapted to local llms, vlm, gguf such as llama-3.3 Janus-Pro, Linkage graphRAG

2,389 stars+3 this week

AI-powered dating simulator

185 stars+2 this week

A simple NPM interface for seamlessly interacting with 36 Large Language Model (LLM) providers, including OpenAI, Anthropic, Google Gemini, Cohere, Hugging Face Inference, NVIDIA AI, Mistral AI, AI21 Studio, LLaMA.CPP, and Ollama, and hundreds of models.

123 stars0 this week

Find the best models and how to run them locally.

248 stars0 this week

DocMind AI is a powerful, open-source Streamlit application leveraging LlamaIndex, LangGraph, and local Large Language Models (LLMs) via Ollama, LMStudio, llama.cpp, or vLLM for advanced document analysis. Analyze, summarize, and extract insights from a wide array of file formats, securely and privately, all offline.

155 stars0 this week

Local-first service attribution, stale development server detection, AI runtime health monitoring, and open-weight model capacity planning.

108 stars0 this week

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

313 stars-1 this week

A large language model (LLM) plugin for Neovim, provides commands to interact with LLM (like ChatGPT, Copilot, ChatGLM, kimi, deepseek, openrouter and local llms). Support Github models.

480 stars-1 this week

A lightweight vLLM-style inference engine in Rust

154 stars0 this week

Production-grade recipe for DeepSeek-V4-Flash-0731 (284B MoE) on 2x NVIDIA DGX Spark: self-healing 2-node vLLM cluster, reboot-verified, tuned DSpark speculative decoding (~75 tok/s), full benchmarks, OpenAI Codex CLI integration

176 stars0 this week

Machine Learning inference engine for Microcontrollers and Embedded devices

772 stars+2 this week

Download models from the Ollama library, without Ollama

154 stars0 this week

ComfyUI nodes for vision-language models: Qwen3-VL, Moondream 3, Florence-2, SmolVLM2, InternVL, Gemma 3, MiniCPM-V. Plus open-vocabulary detection, SAM2/SAM3 segmentation, video temporal reasoning, GGUF via llama.cpp, and hosted LLM/VLM APIs.

592 stars+1 this week

End-to-end documentation to set up your own local & fully private LLM server on Debian. Equipped with chat, web search, RAG, model management, MCP servers, image generation, and TTS.

849 stars+1 this week

KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

518 stars+3 this week

Flash weight streaming for MLX: run massive models larger than your RAM on Apple Silicon.

142 stars+1 this week

Self-hostable multimodal chat with local LLMs (Ollama/OpenAI): PDF RAG, image chat, and Whisper voice, Streamlit + Docker.

208 stars0 this week

Ruby FFI bindings for llama.cpp to run open-source LLMs such as GPT-OSS, Qwen 3.5, Gemma 4, and Llama 3 locally with Ruby.

104 stars0 this week

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

133 stars0 this week

GGUF implementation in C as a library and a tools CLI program

373 stars0 this week

One-click Windows installer for Z-Image Turbo AI image generation. Optimized for low-VRAM GPUs (4GB+). Features Gradio web UI, automatic setup, and GGUF model support.

125 stars+1 this week

Local-first Telegram AI bot with Ollama tool calling, voice transcription, MinIO storage, SearchApi web search, and readiness checks

114 stars0 this week

A robust and efficient TinyML inference engine.

197 stars0 this week

SpectralQuant: Calibrated Eigenbasis Rotation and Water-Filled Bit Allocation for KV-Cache Compression

204 stars0 this week

CUDA编程练习项目-Hands-on CUDA kernels and performance optimization, covering GEMM, FlashAttention, Tensor Cores, CUTLASS, quantization, KV cache, NCCL, and profiling.

114 stars0 this week