Versioning Prompts, Skills, and Evals Like Application Code

Production AI systems accumulate invisible software. Prompts decide what the model is allowed to do. Skills bundle those prompts with examples and tool permissions. Evaluations decide whether a change is an improvement or a regression. If those artifacts live in chat history, a shared document, or one engineer's notebook, you do not have a release process. You have folklore.

We started treating this layer the way we treat application code after the second time a "small wording change" altered SQL generation in a reporting assistant. The application version had not changed. The behavior had. Nobody could answer which prompt was live, who approved it, or how to put the previous one back.

This post is the operating model we use now.

Name the artifacts and give them owners

A prompt is a versioned instruction for a single step. A skill is a versioned packet that can include one or more prompts, input and output schemas, tool access, retrieval rules, and examples. An eval is a versioned set of cases plus the scoring method used to judge a skill or prompt.

Each artifact gets an identifier that does not change, a version that does, and an owner who can accept or reject edits. "The reporting assistant" is not an identifier. reporting.interpret-results is. The owner is a person or a team, not "engineering and the business." Shared ownership in practice means no ownership when an answer goes wrong at 4 p.m.

Put them in source control next to the code that calls them

Prompts and skills belong in the repository of the service that uses them, in files the pipeline can read. We prefer plain text or structured files over records that only exist in a vendor console. Consoles are fine for experiments. They are a poor system of record.

A typical layout is dull on purpose: a directory per skill, a manifest that lists model settings and tool scopes, prompt files for each step, examples used at runtime, and an eval suite that is not deployed to production. The application loads a specific skill version. It does not concatenate whatever document happens to be named "final prompt."

Changes follow the same branch and review path as code. A pull request that alters a skill should show the text diff and the eval result against the previous version. If a reviewer cannot see both, the change is not ready.

Promote through environments the way you promote builds

Development is where people try phrasing. A staging environment runs the candidate skill against the eval suite and against a sample of recent production inputs that have been sanitized. Production loads only versions that passed that gate and were approved by the skill owner.

We pin versions. A production service that quietly starts using "latest" will drift when someone edits a file to fix an unrelated prototype. Pinning also makes incident response possible. If Thursday's answers degrade, you roll the skill version back without rebuilding the entire application, unless the application contract changed with it.

Model names are pinned the same way. A provider alias that tracks a moving target is a change you did not review. When a model must be updated, that is a release with its own eval run.

Evals are tests, so they get the same discipline as tests

An eval case has an input, the context the system would have had, and an expected property of the output. Sometimes the expected property is an exact field. Sometimes it is "must cite page X," "must not invent a filter," or "must refuse." Open-ended scoring without examples produces debates, not gates.

The suite should include cases that used to fail. Those are the ones a prompt engineer will otherwise optimize away. It should also include a small slice of live traffic that has been reviewed and added with consent and redaction rules. Synthetic cases alone make skills look better than they perform.

We fail a release on regressions that matter to the job, not on stylistic variance. If the reporting skill starts emitting a different heading but still returns the correct query plan and the same caveats, that is a note. If it drops a required validator, that is a stop.

Store eval results with the build. Six months from now you will want to know whether a new model is worse than the one you pinned in March, and memory will not be enough.

Review changes with the people who live with the output

Engineers can catch a broken schema and an unsafe tool grant. They are not always the right reviewers for tone on a public answer or for a medical program description. The pull request includes those reviewers when the skill speaks in their domain.

The review question is not "do you like this prompt?" It is "does this version still satisfy the contract we published?" Contracts we write down include refusal rules, citation rules, which tools may run unattended, and which outputs require a human before a side effect occurs.

That contract belongs in the skill repository so it cannot diverge from the instructions the model sees.

Log the version with every production answer

Audit trails are incomplete if they record the user question and the model text but not the skill version, prompt hashes, model identifier, and retrieval set. When a content owner asks why last week's answer cited a retired paragraph, those fields are how you reconstruct the event.

Logging also keeps people honest about hotfixes. If the only way to change production behavior is to ship a version, you will see the change. If someone can edit a portal field that the service reads on each request, you will eventually lose the thread.

Treat examples as data that can go stale

Few-shot examples inside a skill are powerful and perishable. They capture a fee table, a page structure, or a ticket category that will change. We date examples and assign an owner to review them when the source system changes. An example that teaches the model last year's intake steps is worse than no example.

The same is true of eval gold answers. When the policy page is updated, the eval updates in the same change set, or the suite starts enforcing the old policy.

What improved once we did this

The immediate benefit was rollback. The larger benefit was conversation quality. Product owners stopped sending prompt edits through chat. They opened changes against a named skill and saw what the edit would break. Engineers stopped being the only people who knew which text was live.

We still experiment in notebooks. Those experiments do not become production by copy and paste. They become production when they have a version, an eval, a reviewer, and a pin.

If the model is part of a business process, the instructions that steer it are part of the software. Version them, review them, promote them, and record them on the answers they produce. The teams that skip that work keep rediscovering the same incident: the application did not change, the behavior did, and nobody can put the previous mind back into the machine.