Treat the dataset as an executable specification

A supervised example does more than supply text. It rewards a particular response to a particular context. A support record that contains an obsolete escalation path therefore teaches obsolete behavior even when its spelling is perfect. Begin with a task contract: supported inputs, expected output structure, evidence available at inference time, and situations requiring abstention. Record source identity, collection time, transformation version, consent or usage status supplied by the data owner, and a group identifier alongside the text.

The group is the unit whose members must remain together. Several messages from one ticket, translated copies of one document, and variants generated from one template may look like independent rows while sharing the answer. A stable group identifier lets a later split preserve this dependence. Keep raw records separately from transformed records so an unexpected evaluation result can be traced to an explicit decision.

Exact deduplication requires an equivalence rule. For ordinary support prose, Unicode normalization, whitespace collapse, and case folding can reveal copied records. That same rule can corrupt source code, identifiers, mathematical notation, or case-sensitive product names. Define normalization by field type, and retain the original text for training if canonicalization exists only to compute an audit key.

Consider two prompts: "Reset password" and " RESET PASSWORD ". A whitespace-and-case key merges them. However, "reset password for admin" should remain distinct because its authorization requirements differ. Hash a canonical representation with a stable algorithm such as SHA-256 when storing large keys. A digest is an identifier, not anonymization: someone who guesses the original text can calculate the same digest. Always inspect examples near a proposed cleaning boundary before applying it globally.

Near deduplication compares approximate representations, such as sets of token shingles. If two documents contain nine distinct shingles each and share eight, their Jaccard similarity is 8 divided by 10, or 0.8. A threshold of 0.8 merges this pair, but that threshold has no universal meaning. Boilerplate can dominate overlap while a single changed number changes the correct answer. A short negation can reverse intent without greatly changing similarity.

Use approximate similarity to produce candidates, then choose a representative or review uncertain clusters. Record both false merges and missed duplicates on a manually inspected sample. A clustering algorithm also matters: transitive connections can join A to C through B even when A and C are dissimilar. The deduplication research linked below motivates controlling repetition, but your task-specific retention policy still needs its own evidence.

A holdout measures generalization only relative to information unavailable during model development. Protect evaluation groups before generating synthetic paraphrases, choosing examples, tuning prompts, or adapting weights. Check exact text overlap, group overlap, and answer-bearing passages, since a copied answer can contaminate a task even when its question differs. A temporal split is appropriate when deployment concerns future cases, provided timestamps describe the relevant event rather than export time.

Suppose 100 held-out tickets have 20 paraphrases in training. Removing only byte-identical rows leaves an optimistic test. Move every related paraphrase out of training, then regenerate the evaluation manifest and document the change. Never delete difficult evaluation cases merely because they depress a score. If a benchmark influenced repeated development decisions, describe it as a development set and obtain a fresh final holdout.

Create a deterministic audit report with counts at each stage and a reason for every exclusion. For an invented collection of 1,200 rows, a schema check might reject 30, exact deduplication remove 70, and protected-group filtering remove another 100, leaving 1,000. These counts are illustrative bookkeeping, not measured model improvements. Specify ordering because a row with multiple problems receives the first matching reason unless you deliberately record all reasons.

Review the retained distribution by task, language, source, length, and outcome. Deduplication can unintentionally erase rare but legitimate procedures, while aggressive quality filters can favor one writing style. Keep rejection samples and transformation versions so reviewers can challenge the rules. The resulting dataset artifact should answer three questions: what behavior will it teach, which records were excluded, and why is the evaluation meaningfully separate?

This deterministic prose-only audit keeps the first canonical prompt and gives protected evaluation text priority. It deliberately omits semantic deduplication. Change the normalization rule before applying this mechanism to code or case-sensitive identifiers.