What Is AI Inference? How AI "Works" After It Has Been Trained

AI inference is the step of running a trained AI model to produce results from new data. Every time a chatbot answers a question, a system reads an invoice, or a bank scores a transaction, that is one inference.

If AI training is the phase where a model learns from data, AI inference is when the model applies what it has learned to handle real requests.

What is AI inference?

AI inference is the process in which a trained model takes new input data, applies what it has learned, and returns a result: an answer, a classification label, a predicted number, or a piece of text.

In business, AI inference shows up in many familiar tasks:

• Internal assistants: when an employee asks "What documents do I need to open a business account?", the model processes the question, looks up the relevant documents and writes an answer.
• Document processing: the model reads images of invoices, contracts or insurance files and extracts the required fields.
• Fraud detection: every card transaction is given a risk score by the model within a few tens of milliseconds, before it is approved.

A model may be trained only a few times, but it is used for millions of inferences. That is why, once AI goes into production, response speed, processing cost and how data is handled all become important factors to consider.

How is AI inference different from AI training?

Training teaches a model from data and updates its parameters; inference uses the trained model to produce answers without changing its parameters. The two phases need different infrastructure and are measured in different ways.

In the AI development lifecycle, training belongs to the model training and evaluation stages. Inference begins when the model is deployed to production and runs continuously through the monitoring and maintenance stage that follows. Explore all 8 stages of the AI development lifecycle, from problem definition to model monitoring and maintenance.

Most businesses do not train foundation models themselves. They use existing models, sometimes fine-tune them on their own data, and then run inference. If an off-the-shelf model does not understand your business yet, read how enterprises move from generic LLMs to domain-specific AI.

GreenNode currently offers two of the three ways to deploy AI inference described below:

• GPU rental: GPU cloud located in Vietnam, for teams that want to deploy and operate models and serving frameworks themselves.
• Serverless Inference: access ready-to-use models through an API on GreenNode MaaS, billed per token, with no GPU infrastructure to manage.

What happens during an AI inference request?

With an LLM, every request goes through three steps: reading the input (prefill), keeping what was read in memory (KV cache), and writing the answer one token at a time (decode). Understanding these steps explains why an AI application is sometimes fast and sometimes slow.

1. Read the input (prefill). The model takes in and processes the prompt, including the question, attached documents and the conversation history provided. The longer the context, the more data there is to process and the longer prefill can take.
2. Keep what was read (KV cache). The computed results for the input are kept in GPU memory, so each new token does not require re-reading everything from the start. When this memory fills up because too many long requests arrive at once, new requests have to queue, even though the GPU is still running.
3. Write the answer (decode). After processing the input, the model predicts and generates the answer one token at a time. With streaming, content is sent to the user as soon as each token is generated, instead of waiting for the full answer.

To evaluate how fast an LLM responds, businesses usually track two metrics:

• TTFT (time to first token): how long you wait before the first word appears. For a voicebot, this is the silence before the bot starts speaking.
• TPOT (time per output token): how quickly the following words come out, meaning whether the answer flows smoothly or stutters.

When users complain that an AI application is slow, ask whether it is slow to show the first word or slow while the words are streaming. The answer tells you whether the problem lies in context length, queueing, or how the model is served.

Common types of AI inference

The same model can be called in three ways, depending on whether users need the result immediately or can wait.

• Real-time (online) inference: results come back as soon as a request arrives, usually within a few hundred milliseconds to a few seconds. Examples: chatbots, call-center voicebots, card transaction scoring. Latency matters most.
• Batch inference: data is collected and processed in one run, with results available after minutes or hours. Examples: summarizing all of a day's files, classifying thousands of complaint emails every night. Cost per result is usually lower because GPUs are kept busy.
• Streaming inference: the model processes a continuous flow of data. Examples: monitoring system logs, cameras detecting incidents on a production line.

For LLM applications, real-time is the most common type, and answers are usually returned as a stream, word by word.

Where does AI inference run, and why does location matter?

AI inference can run on cloud infrastructure, on servers in a company's own data center, or on end devices. Each option differs in latency, data control, cost and operational requirements.

• Cloud: the model runs on the cloud provider's GPUs and is called through an API. Easy to start and scale, but you need to know which country those GPUs are in.
• On-premise: the model runs on servers in your own data center. Maximum control, but you have to buy GPUs and run an operations team.
• Edge / on-device: small models run directly on cameras, phones or POS terminals. This reduces latency and limits data sent to outside servers, but is usually constrained by the device's compute and memory.

For businesses in Vietnam, where AI inference runs matters especially for two reasons:

1. Response speed: the physical distance between users, servers and supporting services can add network latency. So beyond GPU performance, businesses should consider where the infrastructure sits and the network path when evaluating how fast an AI application responds.
2. Privacy and data sovereignty: prompts can contain personal data, customer information or sensitive business data. If that data is sent abroad for processing, businesses need to assess their legal obligations around cross-border data transfer, and review the provider's policies on data storage, access and use.

Read more: why it matters where AI processes your data.

3 ways businesses deploy AI inference

Businesses can run inference themselves on rented GPUs, use a managed inference service, or call models through a serverless API; the difference is who takes care of which part. The GPU is only the bottom layer. Above it sit the model-serving framework (such as vLLM or SGLang), the KV cache and the model itself, and this is where most speed problems arise.

When comparing AI inference providers, look beyond price and speed and ask: whose GPU runs my prompt, where is it located, and if it gets overloaded, are requests moved to another party? Our comparison of the top AI inference platforms in 2026 goes deeper into this question.

What drives the cost of AI inference?

Inference cost is not just the list price per token or per GPU-hour; it depends on how your application uses the model. Five factors matter most:

1. Input and output tokens. Tokens are the units a model reads and writes; longer answers cost more.
2. Context length. Every time you send long documents or a long conversation history, the model has to read all of it again.
3. Cache rate. If many requests share the same beginning (the same system prompt, the same document set), that part can be cached and does not need to be recomputed.
4. Concurrent requests. Peak hours, not average traffic, decide how many GPUs you need.
5. Model size and GPU type. Larger models need more GPU memory; a smaller fine-tuned model can sometimes deliver similar results at lower cost.

AI inference use cases in business

• Banking and finance: process lookup assistants for staff, call-center voicebots, transaction risk scoring.
• Insurance: reading and extracting information from claim files, classifying requests.
• Healthcare: summarizing medical records into templates, helping read test results.
• Retail and e-commerce: product recommendations, customer service chatbots, generating product descriptions.

Many of these applications have one thing in common: the model may receive customer data or sensitive business information. So beyond model quality, businesses need inference infrastructure that meets their requirements for performance, security and data governance.

FAQ

What is AI inference in simple terms?

It is the moment a trained AI model is put to work: it takes new data and produces a result, such as an answer, a label or a prediction.

Does AI inference need a GPU?

Small models can run on CPUs. LLMs and image models usually need GPUs to reach acceptable speed, because all model parameters and the KV cache have to fit in GPU memory.

Is inference more expensive than training?

A single inference is far cheaper than a training run. But inference runs continuously for every user, so once a model is in production, total inference cost often exceeds training cost.

What is LLM inference?

It is inference with a large language model: the model reads the prompt and writes the answer one token at a time. Two common metrics are TTFT (time to first token) and TPOT (time per output token).

What is an AI inference provider?

A provider that operates GPUs and model-serving software, and lets businesses call models through an API or a dedicated endpoint instead of building the infrastructure themselves.

Next step: Test and compare models on the GreenNode MaaS Playground. If your team wants to run models itself on infrastructure located in Vietnam, contact GreenNode about GPU rental.