vLLM

I analyzed the performance bottlenecks that appeared when concurrent requests increased on an Ollama-based LLM server, then moved the serving layer to vLLM so it could stably handle around 50 concurrent users.

A record of setting up the conditions needed to run the Code Place AI assistant on an operations cluster, including the NVIDIA driver, NVIDIA runtime, k3s containerd configuration, RuntimeClass, and device plugin.