Prompt, Retrieve or Fine-Tune? Choosing the Right AI Approach Inside a Custom Application
One of the least understood decisions in an AI project is also one of the most important: how the application will actually get its intelligence. Teams often jump to the most impressive-sounding option, such as training a model, when a simpler approach would be cheaper, faster and more reliable.
This article explains the main approaches in plain language, compares their costs and trade-offs and offers a way to choose among them. You do not need to be an engineer to follow it, and the vocabulary will help you ask better questions of any development partner.
The Spectrum of Approaches
Think of the options as a ladder. Lower rungs are simpler and cheaper. Higher rungs offer more control but require more data, expertise and upkeep.
- Prompting an existing model. You send instructions and context to a ready-made model and use its response.
- Retrieval-augmented generation. The system first searches your own documents or data, then gives the relevant passages to the model so its answer is grounded in your information.
- Fine-tuning. You adapt an existing model using your own examples so it behaves in a particular style or performs a specialised task better.
- Training a model from scratch. You build a model on your own data. This is rare, costly and usually unnecessary.
A well-run project climbs only as far as it needs to.
Rung 1: Prompting an Existing Model
Prompting means writing clear instructions, perhaps with examples, and sending them to a model through a service. It is the fastest way to build a working feature.
Good for: drafting, summarising, classifying text, extracting simple fields, brainstorming and general-purpose assistants.
Strengths: quick to build, low upfront cost, easy to change.
Limitations: the model knows nothing about your private information unless you include it each time, and it may produce plausible but incorrect statements. Results can vary, so evaluation matters.
Rung 2: Retrieval-Augmented Generation
Retrieval-augmented generation, often shortened to RAG, adds a search step. When a user asks a question, the system looks up relevant passages in your documents, policies or databases and includes them in the request. The model then answers using that material.
Good for: internal knowledge assistants, customer support grounded in your help content, policy and contract questions, and anything where answers must reflect your latest information.
Strengths: keeps answers tied to your sources, allows citations, and updates instantly when documents change without retraining anything.
Limitations: quality depends on how well documents are organised and searched. Poorly structured or outdated content produces poor answers. Access controls must ensure that users only retrieve what they are permitted to see.
For many business applications, this rung delivers most of the value at modest cost, which is why it appears so often in custom projects.
Rung 3: Fine-Tuning
Fine-tuning adjusts a model using examples you supply, so it learns a consistent style, format or specialised task.
Good for: matching a precise tone or output format, handling specialised terminology, improving accuracy on a narrow, repetitive task or reducing the length of instructions needed.
Strengths: consistent behaviour, potential cost savings at high volume, and sometimes better performance on specialised tasks.
Limitations: requires a good set of high-quality examples, careful evaluation and ongoing maintenance when the base model or your needs change. It is not the best way to teach a model new facts, since retrieval handles that more reliably.
Rung 4: Training From Scratch
Building a model from the ground up demands very large datasets, specialist expertise and significant computing resources. It makes sense for a narrow set of organisations, such as those whose core product is the model itself or those with unusual data types that existing models handle poorly. For most businesses, it is the wrong starting point.
Classic Machine Learning Still Matters
Not every intelligent feature needs a language model. Forecasting demand, scoring risk or detecting anomalies in structured data often works best with compact, classic machine learning models. They are cheaper to run, easier to evaluate and more predictable. A thoughtful partner will use the simplest suitable tool, not the most fashionable one.
A Comparison at a Glance
| Approach | Typical build effort | Needs your data? | Best for | Main watch-out |
|---|---|---|---|---|
| Prompting | Low | Optional | General language tasks | Inaccurate or inconsistent output |
| Retrieval (RAG) | Moderate | Yes, documents | Answers grounded in your content | Content quality and access control |
| Fine-tuning | Moderate to high | Yes, labelled examples | Specialised style or task | Data preparation and upkeep |
| Classic ML | Moderate | Yes, structured data | Prediction and classification | Needs clean historical data |
| Training from scratch | Very high | Yes, large volumes | Unusual or core-product needs | Cost and expertise |
How to Choose
Work through these questions.
- Does the task need your private or changing information? If yes, retrieval is likely required.
- Is the task mostly about style, format or a narrow specialised skill? If yes, consider fine-tuning after you have tried prompting and retrieval.
- Is the data structured and the goal a number or category? Consider classic machine learning.
- How costly is an error? Higher stakes call for grounding, evaluation and human review regardless of approach.
- What are the volume and latency needs? High volume may justify a smaller or fine-tuned model to control cost.
- Where may your data go? Privacy or regulatory limits may require specific hosting arrangements.
For a broader view of how these building blocks sit inside a complete product, this overview of bespoke AI software development is a helpful companion.
Evaluation Matters More Than the Approach
Whichever method you choose, you need a way to measure whether it works. Build a set of realistic test cases, including awkward ones, and define what good looks like before development. Test regularly, because models, data and user behaviour change. Without evaluation, you are guessing.
Also include human feedback. A simple button that lets users mark an answer as helpful or wrong provides valuable data for improvement.
Cost and Control Considerations
Using a third-party model service means ongoing usage fees and dependence on the provider’s availability and terms. Self-hosting an open model offers more control and potentially lower cost at scale, but needs infrastructure and expertise. Ask your partner to explain the long-term cost per use, what happens if a provider changes pricing or retires a model and how easily you could switch.
Common Mistakes
- Jumping to fine-tuning too early. Try simpler approaches first.
- Expecting retrieval to fix messy documents. Organise and clean your content.
- Skipping evaluation. A demo is not evidence.
- Ignoring permissions. A knowledge assistant must respect who is allowed to see what.
- Choosing based on hype. Match the method to the problem.
Final Thoughts
The best AI approach is the simplest one that meets your accuracy, cost and risk requirements. Start by prompting, add retrieval when answers must reflect your information, consider fine-tuning for specialised behaviour and reserve training from scratch for rare cases. Measure results honestly, plan for upkeep and choose a partner who explains trade-offs instead of selling the most complicated option.