What LLM deployment includes
You can start with one model and a web chat for the team, and add the gateway, RAG and monitoring as usage grows.
Sizing and hardware
How much GPU memory you need depends on the model, quantization, context length and number of concurrent users.
- ✓VRAM estimate for the model and load
- ✓own hardware or rented GPU server
- ✓comparison with cloud API pricing
- ✓payback period at your volumes
Inference
The engine is chosen for the job: many users, modest hardware or a quick start.
- ✓vLLM for load and parallel requests
- ✓llama.cpp for CPU and small GPUs
- ✓model and quantization chosen for quality
- ✓autostart and restart on failure
API gateway
One endpoint for all applications, like OpenAI, but inside your network.
- ✓LiteLLM with an OpenAI-compatible API
- ✓keys per team and service
- ✓rate limits and token accounting
- ✓fallback to a cloud model when the local one is not enough
Web chat for employees
So that people, not only programs, can work with the model.
- ✓Open WebUI with corporate single sign-on
- ✓chat history on your server
- ✓per-model access rights
- ✓file uploads in chat
RAG over documents
The model answers from your policies, contracts and knowledge base, with links to sources.
- ✓ingesting PDF, DOCX, Confluence
- ✓vector store: pgvector or Qdrant
- ✓scheduled index updates
- ✓answer quality checks on your own questions
Monitoring
A GPU server is expensive, so you need to see how busy it is and where it hits limits.
- ✓GPU utilization, temperature, VRAM
- ✓request queue and response time
- ✓tokens per user and team
- ✓alerts to Telegram or Slack
Security
A private deployment only makes sense if it is actually private.
- ✓access only from the internal network or VPN
- ✓no outbound connections from the model server
- ✓request log for audits
- ✓encrypted disks for documents
Support
New open models come out every few months and are often noticeably better than the previous ones.
- ✓testing new models on your tasks
- ✓zero-downtime model updates
- ✓OS and driver updates
- ✓backups of configuration and indexes
LLM deployment pricing
I give an estimate in hours before starting; work begins after you approve it. Hardware or GPU rental is paid separately.
| Format | Price | What's included |
|---|---|---|
| Assessment | from $230 | tasks, model, hardware, comparison with cloud API and payback period; a report with a plan |
| Deployment | $35/hour | inference, gateway, web chat, RAG, monitoring |
| Support | from $300/month | model and system updates, monitoring, incident response |
Estimate examples
| Task | Estimate |
|---|---|
| Model and web chat for a team on one GPU server | 8-12 hours |
| OpenAI-compatible gateway with keys and limits | 4-6 hours |
| RAG over a document base of up to several thousand files | 16-30 hours |
| GPU and request queue monitoring with alerts | 3-5 hours |
How deployment works
- 1
Tasks and volumes
We work out what the model will do, how many people and programs will use it, and how many tokens you spend per month today.
- 2
Model trial
I run several open models on your examples on a rented GPU before any hardware is bought. That shows whether the quality is good enough.
- 3
Hardware
Based on the trial: a server configuration or rental plan, with a payback estimate.
- 4
Deployment
Inference, gateway, web chat and monitoring. Configuration lives in Git, so the server can be rebuilt from scratch.
- 5
Handover
Docs for the team and administrators. After that you can run it yourselves or leave support to me.
When you need your own LLM
Security or compliance rules out cloud AI
Personal data, trade secrets or client contracts do not allow sending documents to an external provider.
API bills grow every month
Cloud APIs charge for every token. At steady high volumes, your own server pays for itself within months.
Employees already paste documents into public chatbots
Company data ends up in personal accounts. An internal chat covers the need without leaks.
Provider terms and limits change
Rate limits, model deprecations and price changes disrupt processes. A local model keeps working as long as your server does.
When your own server beats a cloud API
Your own server is a fixed cost, an API is a charge per token. Rules of thumb:
- A rented GPU server pays off when your API bill is steadily above its monthly price.
- Buying hardware makes sense with constant load over a year or more; otherwise renting is cheaper.
- Resellers and regional clouds often add a markup on top of the model developer's prices, which shortens the payback.
- If token volume is low or you need the strongest model available, a cloud API is still the better choice.
A hybrid often wins: routine tasks on the local model, hard ones routed through the gateway to a cloud API. For applications it is the same endpoint.
What server do you need for an LLM
The main constraint is GPU memory: it has to hold the model weights and the context of all concurrent requests.
| Model | VRAM (4-bit) | Good for |
|---|---|---|
| 7-14B parameters | 8-16 GB | classification, data extraction, simple chat |
| 30-35B | 24-48 GB | team chat, RAG over documents, code |
| 70B and up | 48-96 GB | complex answers, many users |
| MoE models (Qwen, DeepSeek) | depends on expert count | large-model quality at mid-size speed |
These are rough numbers: each concurrent user with a long context needs a few more gigabytes. I size it precisely for your load.
Ollama, vLLM or llama.cpp
| Ollama | vLLM | llama.cpp | |
|---|---|---|---|
| Strength | easy start | many parallel requests | modest hardware, CPU |
| Load | a few users | tens to hundreds | a few users |
| Hardware | GPU or CPU | NVIDIA or AMD GPU | anything, including CPU |
| Production | for a pilot | yes | yes, for light load |
Ollama is handy for trying things out. For teams and applications I use vLLM, and llama.cpp on modest hardware.
Technologies
- Inference
- vLLMllama.cppOllama
- Models
- QwenDeepSeekLlamaMistralgpt-oss
- Gateway and UI
- LiteLLMOpen WebUI
- RAG
- pgvectorQdrant
- Infrastructure
- DockerKubernetesNVIDIA Container ToolkitROCm
- Monitoring
- PrometheusGrafanaDCGM exporter
Case studies
ML platform: infrastructure and delivery into a closed network
ML services ran on three home servers, and updates had to be delivered into the client's internal network, closed to the outside.
Moved the platform with MLflow to Selectel, set up delivery into the client network via VPN and Git mirroring, monitoring and backups.
Autonomous tractors: ML pipeline on AWS
Perception models for an autonomous tractor fleet needed training and delivery infrastructure.
Set up an ML pipeline on SageMaker, an EKS cluster and deployment via ArgoCD.
Why work with me
Infrastructure, not slides
I don't sell AI transformation. I build a server that works, counts tokens and stays up.
Honest numbers
If a cloud API is cheaper at your volumes, I will say so.
Trial before purchase
The model is tested on your examples on a rented GPU; hardware is bought afterwards.
Everything in code
Configuration lives in Git, with no manual setup nobody can repeat.
Frequently asked questions
How much does a self-hosted LLM deployment cost? +
A model with a web chat for a team on one GPU server usually takes 8-12 hours at $35/hour. Hardware and payback assessment starts at $230, support from $300 a month. The server or GPU rental is paid separately.
Is a local model as good as ChatGPT or Claude? +
No, the strongest cloud models are better. But for working with documents, data extraction, knowledge base answers and coding help, open models are usually enough. This is checked on your examples before buying hardware.
What server do I need for an LLM? +
For a 30B-parameter model and a team of 10-20 people, one GPU with 48 GB of memory is enough. Small models fit in 16-24 GB. The exact configuration depends on the model and the number of concurrent requests.
Does this help with GDPR? +
The model runs on your server and sends data nowhere, so no personal data goes to a third-party processor. The rest of the requirements (access control, logs, retention) apply as to any system that handles personal data.
Can I rent a GPU instead of buying one? +
Yes, for a pilot and variable load renting is cheaper. Own hardware pays off with constant load.
Will our applications written for OpenAI work? +
Yes. The gateway exposes an OpenAI-compatible API, so the application only changes the URL and key.
Send a request
Tell me what tasks you want the model to handle, how many people will use it and how much you spend on APIs now. I'll tell you whether your own server pays off and which one.