jundot / omlx

What it does

• Hot tier (RAM). Frequently accessed blocks stay in memory for fast access.
• Cold tier (SSD). When the hot cache fills up, blocks are offloaded to SSD in safetensors format. On the next request with a matching prefix, they're restored from disk instead of recomputed from scratch - even after a server restart.
• LRU eviction. Least-recently-used models are evicted automatically when memory runs low.
• Manual load/unload. Interactive status badges in the admin panel let you load or unload models on demand.
• Model pinning. Pin frequently used models to keep them always loaded.