A Public Website Copilot That Can Be Cited, Permissioned, and Measured

Internal knowledge assistants and public website copilots get described with the same screenshots. Both have a prompt box. Both retrieve text. Both answer in complete sentences. The similarity ends there.

An internal assistant serves people who already have accounts, training, and a way to complain when the answer is wrong. A public copilot sits on the same page as your phone number and your reputation. It will be asked questions you did not anticipate. It will be quoted in emails. It will be tested by people who are frustrated, curious, or looking for a cheaper path than calling.

We will build a public copilot when the site's search and navigation cannot keep up with how visitors actually ask for help. We will not ship one that only sounds helpful. The design work is about citations, permissions, refusal, and measurement.

Decide what the copilot is for

A public copilot needs a job description that is narrower than "answer questions about our organization." Useful jobs we have shipped or scoped include finding a service and the next step to use it, explaining a published policy in plainer language, helping a visitor choose the right form, and locating a document that already exists on the site.

Jobs we decline for a first release include medical or legal advice, account-specific status unless the visitor is authenticated, comparisons that invent competitor claims, and anything that requires reading a record the website is not allowed to show.

If the team cannot name the job and the questions that are out of bounds, the model will improvise. Improvisation on a public site is how you get confident answers about hours that changed last month.

Retrieval is a publishing problem

The copilot can only be as current as the pages and files you allow it to read. That means the information architecture has to be fit for retrieval, not only for human browsing.

We inventory candidate sources and mark each one as allowed, allowed after cleanup, or excluded. Brochures that predate a program change get excluded. Pages that mix current services with campaign language get split or rewritten. Documents that exist only as scanned images get a text layer or they stay out of the index.

Each retrievable item needs a stable title, a last-reviewed date, and an owner. When those fields are missing, citations look sloppy and nobody knows who should fix a bad answer. The index must drop unpublished and replaced pages on a schedule the business can live with. A CMS publish that takes a week to reach the copilot is a visible defect.

Permissions still apply on a public page

"Public website" does not mean every file in the content tree is fair game. Drafts, staff manuals, pre-decisional memos, and authenticated portal content have no place in an anonymous index. If the site has gated sections, the copilot has to respect the same gate.

We run the retriever under an explicit identity: anonymous visitor, authenticated customer, or staff. Those identities get different corpora. The interface should not imply that signing in will magically produce personal case details unless that path is real and audited.

A related issue is third-party content you display but do not own. If a page quotes another agency or a vendor, the copilot should attribute that material or leave it out. Visitors treat the answer as yours.

Citations are part of the answer, not a footnote you hope nobody opens

A public answer that cannot show its sources is an opinion. We require the response to name the pages it used and to link to them. If retrieval did not return an adequate source, the copilot says it does not have that information and points the visitor to a form, a phone number, or a search results page.

We also cap how far the model may paraphrase a rule. Eligibility, fees, deadlines, and clinical or legal conditions stay close to the source wording. The model can introduce the topic and outline next steps. It should not "simplify" a requirement into something easier to meet.

When legal or communications teams ask whether the copilot can be defended, this is the mechanism. You can show the prompt constraints, the retrieved passages, and the page that was current at the time. That is a weaker standard than a signed letter from counsel, and it is much stronger than "the AI said so."

Refusal is a feature

Teams worry that "I don't know" will feel unhelpful. On a public site, a wrong answer is worse. We define refusal categories in advance:

• the question is outside the published corpus
• the sources disagree and no owner has reconciled them
• the question asks for personal case handling
• the question asks the system to guess a future decision
• confidence or retrieval scores fall below the threshold you set

The refusal should still be useful. Offer the closest relevant page, a human contact, and, when it exists, a form that starts a real request. Track those handoffs. If a large share of questions refuse on the same topic, you have a content gap, not a model gap.

Measure behavior that the business already understands

Page views on the copilot widget are not a success metric. We look at whether visitors who use it reach a qualified next step more often than visitors who only search or browse: form starts, appointment or intake actions, document downloads that correspond to the question, and reduced repeat queries on the same session.

We also read failure. We sample answers weekly and score them against the cited page. We count corrections from staff who were contacted after a visitor arrived with a bad paraphrase. We watch for sudden spikes in a question that the site cannot answer, because that is often a program change the content team has not published yet.

If those measurements are not in the launch plan, the copilot will be judged later by whoever heard the loudest complaint.

A launch shape that stays maintainable

The implementations that last share a few traits. The first release covers one audience and one cluster of tasks. Content owners meet the operating team on a fixed cadence to review sampled answers and stale sources. Prompt and retrieval changes ship through the same review path as other production releases. There is a switch to disable the widget without taking the website down.

None of that is glamorous. It is what keeps a public copilot from becoming an unsupervised spokesperson.

A website copilot is worth building when it helps a visitor use information you have already stood behind. Navigation, search, and staff still have to work without it. If you can cite the answer, restrict what it may read, and measure whether it helped someone complete a real step, it belongs on the site. Until those pieces exist, keep it in a prototype environment.