Support drafts from current policy

Scope and assumptions

The system covers drafting replies for billing tickets. It starts when an agent opens a ticket with the assistant and ends when the agent sends, discards, or rewrites the draft. It never sends by itself and never grants credits.

Illustrative figures assume 4,200 billing tickets a month; 4% of similarity-only drafts would draw on a superseded source (≈170); 6% of drafts would be opened on one ticket revision and sent after the customer wrote again (≈250); about 7% of drafts would contain at least one claim no eligible source supports (≈300).

Architecture

policy wiki (approved versions) ──► indexer: chunks + metadata (version, status, plans, regions, effective dates)
help center ────────────────────┘                     │
                                                      ▼
ticket (revision N) ──► context builder ──► filtered retrieval (eligible only) ──► draft model
                                                                                  │
                                                          claims + links to chunks
                                                                                  ▼
                                                         claim checker (code + model)
                                                                                  │
                                                     draft bound to {versions, revision N}
                                                                                  │
                                              agent edits ──► pre-send check ──► send / stale → rebuild

Source metadata

Each chunk carries: document ID, version, status (draft / approved / superseded), effective-from and effective-to timestamps, applicable plans, applicable regions, owner, and section anchor. The indexer refuses to index an approved document without an effective date and plan scope. Superseding a version sets effective_to on the old one in the same operation.

Retrieval applies these as hard filters first (status = approved, now within the effective window, plan and region match the ticket's account) and ranks by similarity only inside that set. I would use Postgres with pgvector, so filters and vector search run in one query and the metadata is transactional with the policy approval.

Draft and claim format

The model returns structured output: the reply text, and a list of claims. Each claim has the sentence, the chunk IDs it relies on, and the quoted span. The claim checker then:

1. rejects any chunk ID not in the retrieved, eligible set (code);
2. checks that the quoted span exists verbatim in that chunk (code);
3. asks a second, cheaper model call whether the span supports the sentence, with three answers allowed: supports, contradicts, not enough (model);
4. removes claims that fail, and appends a short "not covered" note to the draft.

Steps 1 and 2 are exact. Step 3 is a judgement and is logged with its reasoning for evaluation.

Pre-send check

When the agent sends, the help-desk integration calls the check with the draft ID. It compares:

• the ticket's current revision with the bound revision;
• each bound chunk's version with the current approved version for that document;
• the account's plan and region with those at drafting time.

Any difference marks the draft stale. The agent sees why ("customer replied at 09:13", "policy v5 approved at 11:40") and one button to rebuild. If the check itself fails (service down), the send is allowed with a visible warning, because blocking all support replies is worse than one unchecked draft; that event is logged and counted.

Failure handling

Evaluation

A labelled set of 500 historical billing tickets, replayed with the policy snapshot of their date. Each draft gets one of four labels from a QA reviewer. The key metrics are the rate of unsupported material claims (should approach zero), the stale-source rate (should be zero by construction; any occurrence is a metadata bug), and useful abstention: the share of abstaining drafts where the reviewer agrees no current policy answered the question.

I would not use "agent sent without edits" as a quality signal. Busy agents send plausible drafts; that is the failure we started with.

Trade-offs I considered

• Delete superseded articles instead of filtering them. Simpler, but loses the history QA needs, and help-center articles often lag the wiki anyway.
• Long context instead of retrieval. Putting the whole policy into every prompt avoids ranking errors but not version errors, and it costs more per ticket.
• Fine-tune on past replies. Past replies encode past policy. That is the opposite of what we need.

Stack

Help-desk app integration (sidebar app and a send hook), Postgres with pgvector for chunks and metadata, a hosted model with structured output in an EU region with zero retention, a smaller model for claim checks, and a small admin view for billing operations to see which drafts each policy version affected.

Security and operations

Ticket text contains customer data. It is sent to the model only with the account fields needed for eligibility. Logs store claim-check results and IDs, not full ticket bodies. The metric to watch weekly is the abstention rate per policy area: a rising rate usually means a policy gap, not a model problem.