Foundation models are trained to handle many kinds of tasks for many kinds of users. But they don't automatically understand a company's internal data, industry terminology, processes or rules. That is why a chatbot can answer smoothly in a demo and still make mistakes on real files.
Not every problem needs fine-tuning. Sometimes better prompts are enough, sometimes the model needs to be connected to company data through RAG, and sometimes you need to fine-tune or deploy your own AI model to meet business requirements.
MIT NANDA's The GenAI Divide 2025 report found that around 95% of organizations have not yet seen measurable returns from GenAI. The main reason was not infrastructure or talent, but that systems "do not retain feedback, adapt to context, or improve over time" (Virtualization Review). In other words, the model has not been brought into the business.
This article helps you answer three questions: what foundation models are missing, how to tell when your company has hit that limit, and when you really need to fine-tune or build your own AI model.
5 reasons foundation models fall short on business-specific tasks
1. The model doesn't know your internal data
Foundation models don't automatically have access to a company's own data, such as fee schedules, product catalogs, contract templates or case-handling history.
When the information it needs is missing, a model can still produce an answer that sounds plausible but is wrong. This is especially risky when AI is used to process documents, look up policies or support business decisions.
To fix this, companies can add the information to the prompt or connect the model to internal data sources through RAG, so the model retrieves relevant documents when it answers.
2. Industry terms and phrasing differ from everyday language
Every industry has its own terminology, business codes and ways of saying things. Banks talk about "group 2 debt", hospitals use ICD codes, and companies use abbreviations that only make sense internally.
A foundation model may know the everyday meaning of a term but not interpret it correctly in a professional context. For Vietnamese and other Southeast Asian languages, performance also depends on how well these languages are represented in the training data.
If the model keeps confusing terms even when given enough information, companies can consider fine-tuning on real business examples or choosing a more specialized model.
Learn more about the gap between general-purpose LLMs and specialized AI in From Generic LLM to Domain-Specific AI.
3. Processes and decision rules are tacit knowledge
Much business knowledge does not live in documents; it builds up through experience handling real cases.
For example, bank staff need to know when a loan application must be escalated for approval, which cases qualify for exceptions, and how to prioritize complaints. These decisions can depend on many conditions and on practical experience.
A foundation model does not understand these rules by itself. Companies need to provide guidelines, processes and suitable worked examples. For repetitive tasks, fine-tuning can help the model learn to perform them more consistently from reviewed examples.
4. Output must be in the right format and consistent, not just "roughly right"
In a business setting, an answer with the right content may still not meet the requirements of downstream systems. Data may need the correct JSON structure, the correct product codes, the correct fields or brand standards.
Foundation models sometimes produce inconsistent output from one run to the next, even for similar requests. At scale, these small deviations increase manual review work and make automation harder.
Companies can start by standardizing prompts and validating outputs against the required structure. If the model still often gets the format wrong or ignores instructions, fine-tuning on standard input and output examples may be the right approach.
5. Data is not allowed to leave the company
Some foundation model services require companies to send prompts and input data to the provider's infrastructure through an API. This may conflict with security policies, data governance requirements or regulations in sectors such as banking, insurance and healthcare.
Companies need to assess where data is processed, how it is stored, who can access it, and how the provider uses it. Depending on their requirements, they can choose a service with suitable commitments or deploy models on infrastructure they control.
When business data cannot be sent to outside AI systems, the ability to customize the model can also be limited. However, fine-tuning does not automatically solve data sovereignty. Companies still need to control how models are trained, stored and deployed.
Signs that foundation models no longer meet your company's needs
If your AI project shows two or more of the signs below, review how data and business rules are brought into the system, instead of simply switching to a bigger model.
- Great in the demo, weaker in real operations. A pilot with sample questions performs well, but accuracy drops on real files, emails or documents.
- Prompts keep getting longer, but errors keep coming back. The team keeps adding rules to the prompt, raising token costs and latency without fully fixing business errors.
- Staff still have to check almost every output. Time saved by AI is eaten up by reviewing, correcting and re-entering data.
- The model gets critical details wrong. It writes fluent answers but confuses product names, business codes, figures or contract terms.
- Legal or information security has not approved the project. Where data is processed, how it is stored, or the service's terms of use do not meet internal requirements.
- API costs rise but quality doesn't improve accordingly. When scaling from 10 to 1,000 employees, costs can rise sharply while core business errors remain.
These signs don't necessarily mean you have to build your own model. The next step is to identify whether the cause lies in knowledge, model behavior, data quality or how the system is deployed.
You don't always need to fine-tune: choose the right way to "teach" the model
There are four ways to bring business knowledge into a model, from lightest to heaviest: write better prompts, connect the model to data (RAG), fine-tune, and build your own AI model. OpenAI's Optimizing LLM Accuracy guide separates two kinds of problems: a model that lacks knowledge needs more context (RAG); a model that behaves inconsistently (wrong format, wrong tone, applies rules unevenly) needs fine-tuning. The two can be combined.
| Problem | Suitable approach | Why |
|---|---|---|
| Generic answers, lacking clear instructions | Prompt engineering | Fast, low-cost and easy to experiment with |
| Model lacks information on products, policies or new documents | RAG | Retrieves relevant information without retraining the model |
| Wrong terminology, format or inconsistent tone | Fine-tuning | Teaches the model how to perform the task from standard examples |
| Repetitive business rules that must be applied consistently | Fine-tuning (with RAG if lookups are needed) | Improves how tasks are handled and retrieves information when needed |
| Need control over infrastructure and data, or lower cost at scale | Your own AI model, self-operated | More control over the model and its operating environment |
A practical rule: start with the lightest step and measure it with a test set drawn from real work. Only move up a level when the current one stops improving. Adding RAG to a behavior problem can even make results worse, because the extra context becomes noise.
You can start the first two levels right away with foundation models: Serverless Inference on GreenNode MaaS lets you test and compare models on your own data and prompts before deciding whether you need to fine-tune. When you need to run your own model, you can rent GPUs located in Vietnam instead of investing in hardware.
When does a company need to fine-tune or build its own AI model?
A company needs to fine-tune when prompts and RAG are already done well but the model still gets terminology, format or business rules wrong. It needs its own AI model when data cannot leave its infrastructure, or when volume is high enough that running a model itself is cheaper than calling an API. Three conditions should be in place before you start:
- A specific, repetitive task. For example: classifying complaints, extracting information from insurance files, summarizing medical records into a template. The narrower the task, the easier it is for a small fine-tuned model to beat a large general-purpose one.
- Good example data. A few hundred to a few thousand input/output pairs reviewed by people who do the work. Quality matters more than quantity, and the examples should look like the real data the model will see in production.
- A clear way to measure. A test set and a "pass" criterion agreed by the business team, so the old and new models are compared on the same yardstick.
With methods such as LoRA, fine-tuning a 7B–14B open-source model is no longer a months-long research project. The harder part is usually infrastructure: GPUs for training, secure data storage, model version management, and deploying the tuned model as an API for other systems to call, which is the AI inference step.
Train, fine-tune and run inference on GreenNode AI Platform
GreenNode AI Platform helps companies train, fine-tune and run inference on AI models for their own use cases, on GPU infrastructure located in the region:
- Customization for your business and language: fine-tune on your data so the model understands your industry terms, Vietnamese and other Southeast Asian languages.
- Data sovereignty: train on your own data without sending it abroad.
- The full model lifecycle: GPU-backed Notebooks for fine-tuning, Model Registry for version management, and Inference to turn models into autoscaling endpoints.
- Cost efficiency and local support: small tuned models cost less to run than large general-purpose ones, with an engineering team supporting you in Vietnam.
Want to see how a fine-tuning pipeline works end to end? Read From Generic LLM to Domain-Specific AI.
Next step: Explore GreenNode AI Platform and start with one specific business task to assess how AI can work for your company.
FAQ
Why fine-tune an AI model?
Fine-tuning teaches a model your company's specific behavior from real examples, such as industry terms, output formats and handling rules. It becomes necessary when prompts and RAG can no longer improve accuracy or consistency.
What is a custom AI model for a business?
It is a model, usually built from an open-source model, that is fine-tuned on a company's data and runs on infrastructure the company controls, instead of a shared model called through a provider's API.
What is the difference between RAG and fine-tuning?
RAG adds knowledge by looking up documents at answer time. Fine-tuning changes how the model behaves. If information is missing, use RAG; if the model does things the wrong way, fine-tune.
Is a custom AI model expensive?
With LoRA and 7B–14B models, fine-tuning costs far less than training from scratch. At high volume, a small tuned model is often cheaper to run than calling a large model through an API.
How much data do you need to fine-tune?
Usually a few hundred to a few thousand high-quality examples for a narrow task. Examples reviewed by people who do the work matter more than sheer volume.
