VectorInstitute / vector-inference
Efficient LLM inference on Slurm clusters.
Why this job: says: llm inference; topics: vllm, inference
What it does
This repository provides an easy-to-use solution to run inference servers on Slurm-managed computing clusters using open-source inference engines (vLLM, SGLang). This package runs natively on the Vector Institute cluster environments. To adapt to other environments, follow the instructions in Installation.
Get started
pip install vec-inf- Maintenance
- Active
- Last commit 15 days ago
- Stars
- 111
- +1 this week
- Licence
- Apache-2.0
- Allows business use
- Downloads for
- None listed
- No release files
- Latest version
- v0.9.0
- Released 7 months ago
- Built with
- Python
- 98% of the code
Is it looked after?
- Contributors
- 3
- Last 90 days
- Issues
- 87% closed
- 6 still open
- Pull requests
- 246 merged
- 5 waiting
- Releases
- 21
- 4 in the last year
- Contributing guide (opens in new tab)
- No security policy
- No code of conduct
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| kvcache-ai / Mooncake | 7,006 | today | Apache-2.0 | Linux |
| SemiAnalysisAI / InferenceX | 2,010 | today | Apache-2.0 | No downloads |
| microsoft / vidur | 721 | 2 months ago | MIT | No downloads |
More: languages, topics and project record
Languages
- Python 98%
- Dockerfile 2%
- Shell 1%
Topics
- audio-transcription
- inference
- llm
- llm-infernece
- llm-infrastructure
- multimodal
- reward-model
- sglang
- speech-to-text
- text-embedding
- vllm
- vlm
Project record
- JobLocal LLMs
- StatusListed
- Commits1,402
- Forks14
- Watching7
- Stars in 28 days+1
- CreatedMar 6, 2024
- First seen by usNov 7, 2026
- CheckedNov 7, 2026