Fine-Tune, RAG, or Reusable Skill? A Decision Guide for Mid-Sized Teams

When a mid-sized organization is ready to put language models into a real workflow, the first architecture conversation often starts in the wrong place. People ask which model to buy. The more useful question is how the system should learn the work.

Most production needs we see fall into three patterns. Retrieval-augmented generation, or RAG, gives the model current documents at answer time. A reusable skill packages a known procedure, examples, and constraints so the model performs a task the same way every time. Fine-tuning changes the model's weights so a behavior or voice is available without sending the full procedure on every request.

Those options overlap, and vendors will sell all three at once. For teams without a dedicated machine learning staff, the cost of choosing the wrong one is not theoretical. You either overspend on training data you did not need, or you ship an assistant that sounds confident while reading last year's handbook.

This guide is the sequence we use with clients before anyone labels a folder "training data."

Start with the job, not the technique

Write down the job in one sentence that a staff member would recognize.

• Answer employee questions from current policies and department pages
• Turn a support transcript into a draft knowledge article
• Generate a first-pass security summary from a scan export
• Convert a stakeholder interview into a structured requirements list
• Produce a weekly narrative from an approved reporting view

Then ask four questions about that job.

• Does the correct answer change when a document is edited next week?
• Is there a stable method the team already trusts, with clear inputs and outputs?
• Does the work require a house style or a domain dialect that general models keep missing even when the right text is in context?
• What happens if the system is wrong: embarrassment, rework, or a compliance incident?

Question 1 points toward retrieval. Question 2 points toward a skill. Question 3 is the only one that starts a fine-tune discussion. Question 4 sets the review burden for whatever you pick.

RAG when the facts must stay current

RAG is the default for assistants that answer questions from living content: policies, service pages, manuals, tickets that have been promoted into documentation, and approved reports. The model is not supposed to remember the employee handbook. It is supposed to find the current section and stay inside it.

RAG is the right first investment when:

• documents are already the system of record
• updates happen outside the model
• answers need citations
• different users should see different slices of the corpus
• you cannot freeze the knowledge long enough to train on it

The failure mode is treating RAG as a document dump. Retrieval quality depends on how content is chunked, titled, permissioned, and refreshed. If two versions of a policy are both searchable, the model will reconcile them with invented compromise language. We have written separately about reducing those hallucinations by tightening retrieval and verification. The architecture lesson here is simpler. If freshness is the requirement, put your effort into the corpus and the retrieval pipeline, not into training a model to memorize pages that will change.

RAG is a poor fit when there is no decent corpus. A model cannot retrieve a procedure that exists only in one senior analyst's head. In that case you document the procedure first. Then you decide whether it belongs in retrieval, in a skill, or both.

Skills when the method is stable

A reusable skill is a versioned packet: instructions, constraints, input schema, output schema, and a short set of worked examples. It is closer to a runbook than to a knowledge base. We use skills for jobs like first-pass PageSpeed analysis, structured requirements extraction, scan-finding correlation, and draft SQL that must pass a reviewer before it runs.

Skills earn their keep when:

• the steps are repeatable
• the output shape is more important than open-ended conversation
• the same mistakes keep appearing unless you forbid them
• several applications should perform the task the same way
• you want to improve the work by editing a document, not by retraining a model

A skill can call RAG. That combination is common. The skill says how to analyze a page or a ticket. Retrieval supplies the current reference material. Keep those layers separate in source control so you can change the method without re-indexing, and change the corpus without rewriting the method.

Skills fail when teams hide half the business rules in chat history. If only one engineer knows the "real" prompt, you do not have a skill. You have a personal shortcut. Promote the packet, give it a version, and put a reviewer on changes the same way you would for a stored procedure.

Fine-tuning when behavior will not stick any other way

Fine-tuning is the most expensive option to operate well, even when the training run itself looks cheap. You need representative examples, a way to judge them, a retraining plan when the business changes, and a rollback path if the new weights pick up an unwanted habit.

We consider it only after RAG and skills have been tried against the same examples. The cases that survive that filter usually look like this:

• the model keeps ignoring output contracts even with strong instructions and examples
• the domain language is dense enough that prompting wastes a large share of the context window
• you need a smaller model to run under a data-residency or latency constraint, and that smaller model only becomes usable after task-specific training
• classification or routing is high volume and low creativity, and a tuned model is cheaper than sending a large context every time

Fine-tuning is the wrong tool for keeping facts current. Weights are a snapshot. If a fee schedule changes on Friday, a tuned model will not know unless you retrieve the schedule or train again. Teams that fine-tune on policy manuals are usually trying to avoid building retrieval. They pay for that avoidance the first time legal updates a paragraph.

It is also the wrong first response to a tone problem. If the assistant sounds generic, add style rules and approved examples to the skill. If it still drifts after that, then you have evidence, not a hunch, that weights may help.

A practical way to choose

We walk clients through a short path rather than a matrix with twelve axes.

If the answer must reflect documents that change, start with RAG and a verification step.

If the work is a procedure with a defined output, wrap it in a skill. Add retrieval inside the skill only for the facts that move.

If a skill plus retrieval still fails a held-out set of real tasks, isolate the failure. Missing documents are a corpus problem. Inconsistent method is a skill problem. Persistent format or dialect failure, after those are fixed, is the opening for a fine-tune on that narrow task.

Do not fine-tune a general assistant because the prototype felt uneven. Uneven prototypes are usually retrieval and evaluation problems.

Cost and ownership sit with the same decision

RAG costs you content operations: cleaning pages, removing duplicates, setting permissions, and refreshing indexes. Skills cost you product thinking: writing the procedure well enough that a model can follow it, and versioning the packet. Fine-tuning costs you a data pipeline and a retraining calendar.

Mid-sized teams can staff the first two with the people they already have, if those people are given time. The third needs someone who will own examples and evals after the vendor workshop ends. If that owner does not exist, do not start a fine-tune so the project can look advanced.

What we tell stakeholders

Use RAG to keep the system honest about current facts. Use skills to keep the system consistent about how work gets done. Use fine-tuning only when you can prove that honesty and consistency are not enough.

The unglamorous order is usually the one that reaches production.