When you put an AI model into production, choosing the right model is only half the decision. You also need to decide how to run inference in a way that balances cost, performance and scalability.
There are two common options. The first is Serverless Inference, where you call a model through an API and pay for what you consume. The second is renting dedicated GPUs, where your team deploys and manages the model on GPU resources allocated to you.
Neither option is best for every workload. The right choice depends on your token volume, how much your traffic fluctuates, your performance requirements, how much control you need over data, and your team's operational capacity.
This article compares Serverless Inference and dedicated GPUs across 7 key criteria, shows how to calculate the cost break-even point, and gives you a checklist for deciding when to move between the two.
How Serverless Inference and self-hosting on GPUs differ
Serverless Inference means calling a model that the provider has already deployed, through an API, under the Model-as-a-Service (MaaS) model. You don't manage servers, configure GPUs or worry about scaling infrastructure. You pay for the input and output tokens you actually use. No requests, no cost.
Self-hosting a model on dedicated GPUs (self-hosted / dedicated inference) means deploying the model yourself on a GPU instance rented by the hour or by the month. Your team chooses the model (open-weight or a fine-tuned version), installs an inference server such as vLLM, SGLang or TensorRT-LL, and operates the inference endpoint that serves your application. You pay for the time the GPU is allocated, whether it is processing requests or sitting idle.
In short, with serverless you pay for usage; with self-hosting you pay for reserved capacity. Every difference in cost, latency and control follows from this.
Serverless Inference vs. self-hosting on GPUs: 7 criteria compared
Serverless Inference usually wins on speed to deploy, scalability and cost when traffic is low or fluctuates. Dedicated GPUs give you more control and can lower costs once a workload is large and steady enough.
If you are comparing specific vendors, see the top AI inference platforms in 2026. The table below summarizes the key differences to help you evaluate which option fits.
| Criterion | Serverless Inference | Dedicated GPU |
|---|---|---|
| Pricing | Per input/output token | Per hour or month, including idle time |
| Time to production | Minutes: create an API key and call it | Days to weeks: pick GPUs, build the serving stack, load-test |
| Available models | The provider's catalog of deployed models | Any model that fits on the GPU: open-weight, fine-tuned or in-house |
| Latency | Depends on overall system load; subject to rate limits | More stable since resources aren't shared; batch size and quantization can be tuned |
| Scaling | Automatic, per request | Add GPUs and load balancing yourself |
| Data | Prompts pass through the provider's infrastructure | Prompts and logs stay on machines you control |
| Operational effort | Close to zero | Requires people to monitor, update and handle incidents |
On data, ask your serverless provider exactly where each model runs. Many platforms host some models themselves and route other requests to third-party APIs. The two have different data residency scopes. See also how to choose a MaaS platform in Vietnam and APAC.
The cost question: which is cheaper, and how to find the break-even point
Cost is one of the most important factors when choosing how to deploy AI inference.
However, dedicated GPUs do not automatically become cheaper than Serverless Inference as token volume grows. The cost advantage appears only when a workload is large enough to use GPU capacity efficiently and cover the cost of operating the infrastructure.
To evaluate this, you need to find the break-even point between the two options.
Where:
- T* is the number of million tokens per month at the break-even point
- N is the number of GPUs needed to run the model, R is the hourly rental price per GPU, and 730 is the average number of hours in a month (24 hours × 365 days ÷ 12 months ≈ 730)
- O is the monthly operating cost: engineering time, monitoring, storage
- P is the serverless price per million tokens (blended across input and output at your actual ratio)
You also need to compare T* against the GPU's actual capacity. A GPU running a mid-sized model with continuous batching can only process a certain number of tokens per second. If T* exceeds that capacity, self-hosting this model will never be cheaper.
Illustrative example. The figures below are assumptions used to show the calculation, not any provider's pricing. Suppose you rent 1 GPU at VND 50,000/hour, or about VND 36.5 million/month. The GPU reaches a throughput of 1,500 tokens/second, roughly 130 million tokens/day at full load. Operating cost O is left out for now.
| Serverless price (VND / 1M tokens) | Break-even point (M tokens/day) | Required GPU utilization |
|---|---|---|
| 5,000 | ~240 | ~185%, not achievable |
| 10,000 | ~120 | ~93% |
| 20,000 | ~60 | ~46% |
The table shows three things:
- The cheaper a model is via API, the harder it is for dedicated GPUs to compete on price. For popular open-weight models, serverless providers already optimize batching at scale.
- GPU utilization decides everything. If traffic is concentrated in business hours, the GPU sits idle nearly two-thirds of the time but is still billed.
- Operating costs push the break-even point higher. Adding O to the formula raises T* accordingly. A model that needs 2 or more GPUs multiplies T* as well.
Industry analyses typically put the break-even threshold at 30–60% average GPU utilization, depending on the model and provider. The safest approach is to measure with your own data: run on serverless for a few weeks, collect actual token counts, then benchmark the model's throughput on the GPU you plan to rent.
When to choose Serverless Inference and when to self-host on dedicated GPUs
Once you've assessed cost, look at your workload characteristics and engineering resources to decide which option fits.
Choose Serverless Inference when:
- Your product is at the PoC or MVP stage. If you don't know your user volume yet, don't commit to GPU capacity.
- Traffic is low or highly variable. For example, an internal chatbot used only during working hours, or document processing jobs that run in batches.
- You need to test several models. Switching models only means changing the model name in the API request, with no redeployment.
- Your team has no dedicated MLOps engineer. Engineering time is better spent on the product.
Choose to self-host on dedicated GPUs when:
- Token volume is large and steady. GPU utilization stays high for weeks, above the break-even point you calculated.
- You use your own model. A version fine-tuned on company data (see why foundation models fall short on specific business tasks), a model quantized with a custom configuration, or a model not available on any serverless platform.
- You need consistent latency. Real-time applications need a stable time to first token (TTFT), unaffected by other customers' load and free of rate limits.
- You need tight control over data. BFSI and healthcare organizations often need to know exactly where prompts and logs are stored and who can access them (see why data processing location matters in the AI era).
If your workload falls into both groups, that's a sign a hybrid setup is the better fit.
Combining Serverless Inference and self-hosted GPUs to optimize AI workloads
You don't have to commit fully to one option.
In practice, a hybrid inference architecture can combine dedicated GPUs with Serverless Inference to balance cost, performance and scalability. Most enterprises don't pick just one side. Three common hybrid patterns:
- Dedicated GPUs handle base load, Serverless Inference absorbs spikes. Dedicated GPUs handle the steady base load. Serverless Inference takes the overflow above it. Size GPU capacity to the base load, not the peak.
- Split infrastructure by task type. Real-time tasks run on dedicated GPUs (see low-latency architecture for RAG and AI agents). Batch tasks, where no one is waiting for the result, run on serverless.
- Serverless Inference for experimentation, dedicated GPUs for production. Dev and staging use serverless to test models quickly. Production moves to dedicated GPUs once token volume is large enough.
Before moving a workload from Serverless Inference to dedicated GPUs, check these 7 points:
- Usage data: You have at least 4–8 weeks of token data, including hourly traffic distribution and peak times.
- Real-world performance: You have benchmarked throughput, TTFT and latency on the exact GPU type you plan to use.
- Hardware requirements: You know how many GPUs, how much VRAM and what level of quantization the model needs (e.g. H100 vs H200).
- Total cost: You have included GPU rental, operations staff, monitoring and storage.
- Model quality: You have checked that output quality matches across both environments, including model version and quantization settings.
- Operational capacity: A team is responsible for monitoring, updates and incident response.
- Fallback plan: You have a fallback mechanism for when GPUs are overloaded, under maintenance or failing.
Running Serverless Inference and dedicated GPUs on GreenNode
GreenNode offers both options on the same infrastructure, so you can start small and shift over time without changing providers.
- Serverless Inference (MaaS): call models through an OpenAI-compatible API, pay per token, with no fixed monthly fee. Open-weight models hosted by GreenNode run on infrastructure in Vietnam and Southeast Asia (see the 8 newly launched self-hosted models).
- GPU Instances: rent NVIDIA GPUs on demand (H200, H100, L40S, A40, RTX 5090, RTX 4090) to deploy models yourself, including fine-tuned versions.
For flexibility, GreenNode also offers AI compute for inference without long-term contracts. Suggested path: start with Serverless Inference to measure your actual token volume, then move the base load to GPU Instances once you pass the break-even point.
Frequently asked questions
Does Serverless Inference have cold starts?
For popular models that the provider keeps warm, cold starts are negligible. Cold starts mainly happen on serverless GPU platforms that have to load your model into memory on the first request.
Can I run a fine-tuned model on Serverless Inference?
It depends on the provider. If the platform doesn't support deploying custom models, self-hosting on dedicated GPUs is the most direct option.
When should I recalculate the break-even point?
When token prices change, when you switch to a model of a different size, or when your token volume changes significantly since the last calculation.
Does moving from serverless to self-hosting require major code changes?
If both use an OpenAI-compatible API (vLLM and SGLang both support it), you usually only need to change the base URL and the model name.


