Most B2B AI products in 2026 are an OpenAI bill with a UI on top. We are not. The chat AI, the ticket AI, and the wait-time concierge on phone calls all run on hardware we bought, in our office. Here is the math that made us do it.
Starting point: what we were paying a hosted provider
We started on a hosted inference endpoint. Pricing was roughly $0.15 per million input tokens and $0.95 per million output tokens. Real-world usage: roughly 1.2k input tokens and 320 output tokens per conversation, AI-driven. That is $0.00018 input + $0.00030 output = $0.00048 per conversation. At 100k conversations a month, $48. At 1M, $480.
At 100k per month, hosting is irrelevant. Hosting becomes interesting somewhere between 500k and 2M conversations per month. We crossed that threshold in March.
What the hardware actually cost
A single high-end inference GPU: roughly $9,200 (we got a partner price). Plus the rest of the workstation: CPU, 128GB ECC memory, two NVMe SSDs, a workstation PSU, a quiet chassis, and a UPS. Total all-in: $11,840.
Inference throughput we hit
Running our quantized model on our own hardware, we benchmark at 240 tokens per second sustained for our prompt mix. Concurrency 4 to 6 before queueing matters. With a 50ms p50 first-token latency. That is enough headroom for roughly 800,000 conversations per month before queueing kicks in, with conservative settings.
The cost-per-conversation comparison
“We break even with hosted inference at about 850k conversations a month. We hit that in our 4th month of paid customers.”
Hosted: $0.00048 per conversation, scales linearly. Self-hosted on our own hardware: $11,840 of capex amortized over 36 months is $329 per month of hardware cost. Plus ~$80 per month in electricity. So $409 per month of fixed cost, divided by ~800k conversations, equals $0.0005 per conversation. We break even with the hosted option at about 850k conversations a month.
The non-cost reasons
Cost was the trigger, not the only reason. Three other reasons mattered. (1) Latency. Self-hosted p50 is 50ms first-token. A hosted endpoint runs closer to 280ms. On a chat that runs back-and-forth, the difference is felt. (2) Privacy posture. Customer conversations never leave our infrastructure and are never sent to a third-party model provider. Selling to HIPAA-aligned customers is easier. (3) Roadmap independence. We are not locked to a vendor's quantization choices, model versions, or rate-limit policies.
When self-hosting does not make sense
If you are running fewer than 200k AI conversations a month, you should stay on a hosted endpoint. The math will not work yet. If you are running more than 200k and growing, model it. We were on hosted inference for 7 months and the spreadsheet did not flip until month 6.