A fictional museum tests an AI assistant that drafts short exhibit captions from approved collection records. After staff update one policy and add a new object, a fresh evaluation seems worse than last week’s. Did the model regress, or did the test data change? LangSmith datasets are versioned when examples are added, updated, or deleted, and important versions can be tagged. This enables a controlled comparison, but only when the team also preserves the application’s configuration, expected outputs, and review rules.
Define what the caption evaluation measures
Create cases that test a single approved source record, attribution uncertainty, an embargoed object, conflicting acquisition dates, and a caption that must not invent a maker. Each input should point to an authorized record revision and expected behavior such as ‘state unknown’ or ‘withhold public caption.’ Avoid scoring only whether the output sounds engaging. Fact accuracy, disclosure permissions, accessibility, and refusal to speculate are separate dimensions deserving their own judgments.
Use synthetic or authorized examples and store provenance alongside each case. A case with no verified artist must not have a reference answer that confidently names one. Write down whether the assistant is supposed to draft for a curator or publish directly; this guide assumes staff-reviewed drafts only. Define the intended outcome and a reviewer for every change to ground truth. A model cannot determine which archival revision is legally approved for publication.
Understand how LangSmith versions datasets
LangSmith documentation says adding, updating, or deleting examples creates a new dataset version. Versions can be inspected by timestamp, while previous versions are read-only in the example viewer. That matters when an editor fixes a mistaken reference caption in the middle of an experiment: the suite is no longer identical to the one used earlier. Record the version associated with each test rather than referring vaguely to ‘the museum dataset.’
A fictional example starts with twelve approved caption cases, then adds a thirteenth case for an embargoed ceramic bowl. Comparing a new run on thirteen cases with an old run on twelve can mislead even if all original cases behave identically. Keep the old suite addressable and compare on a matched version first. The numbers in this example illustrate record counts, not a real benchmark or verified LangSmith experiment.
Tag a reviewed baseline
LangSmith supports tagging dataset versions with readable milestone labels, such as ‘prod’ in its documentation. For this project, a team might adopt an internal tag like ‘caption-policy-approved’ after the curator signs off on the expected outputs. A tag should identify an agreed test state rather than imply that the assistant itself is production-ready. Store the relevant date and approving person in the release record, not in generated caption text.
Tags can become confusing if a team silently repoints an important label. Establish whether a tag is expected to remain fixed for an audit or represent a moving current baseline, and record any movement. Preserve the underlying exact version or timestamp alongside the friendly tag. When a reference answer changes for legitimate curatorial reasons, create a review note explaining why the old judgment no longer applies. Never alter historical results to make the new model look better.
Compare model changes on matched inputs
Run the proposed caption prompt against the pinned dataset version before adding new cases. Keep the retrieval index revision, model identifier, system instructions, and evaluator criteria in the experiment metadata. LangSmith can associate experiments with dataset versions, which helps distinguish shifts in application behavior from edits to the tests. Review the same case IDs in both outputs, not just aggregate scores. A single incorrect claim about ownership can matter more than a small score movement.
Compare pairs such as ‘maker unknown’ becoming a fabricated name, a correct accession number becoming a near match, or a label exposing unpublished donor information. Show the source passage and the reviewer’s rationale beside the generated draft. If the new prompt is better on style but worse at withholding uncertain claims, don’t hide that tradeoff under a single quality number. Evaluator feedback needs human calibration for the museum’s actual editorial standard.
Separate changed expectations from real regressions
Sometimes the source-of-truth record changes. Suppose a curator confirms a date previously marked unknown, then updates both the approved record and the test expectation. That is a legitimate dataset revision, not evidence that last month’s model was wrong for following the older source. Compare performance on the old pinned version for model-only changes, and on the new version for readiness under the updated policy.
A three-panel diagram can show a fixed baseline, an uncontrolled mix of cases, and a reviewed new baseline. These are hypothetical configurations, not LangSmith screenshots or actual measured results. In your release note, separate the delta due to model/prompt changes, the delta due to added examples, and the delta due to corrected references. If the dataset is too small for a meaningful numerical claim, provide case-level findings instead of a misleading percentage.
Review privacy, permissions, and example quality
An evaluation dataset can include sensitive records just as a production model prompt can. Do not upload embargoed captions or donor details without confirming the organization’s allowed storage and data processing terms. Prefer synthetic placeholders for demonstrations. Restrict who can edit reference answers, add new cases, or share a dataset publicly. LangSmith offers sharing and export features; that does not mean every museum example should be distributed or made accessible to all project members.
Set a clear process for annotating examples that contain ambiguity and for retiring cases that no longer reflect the active collection. A case should be reproducible through its permitted input revision and expected rule, not through a stale web snippet. When a reviewer marks an output incorrect, store the specific factual or policy violation. That detail makes future fixes more actionable than a generic ‘bad caption’ label, and it protects against optimizing a model for arbitrary style preferences.
Make the final release decision auditable
An approval packet should name the candidate prompt and model, the dataset version or tag, source-record revision, evaluator settings, date, and open failures. A curatorial reviewer should check representative passed cases as well as the failures because automatic scores can miss newly introduced factual assertions. Only an authorized editor should approve a caption for publication. Evaluation success does not grant rights to exhibit an object image or publish a restricted attribution.
After rollout, monitor real editorial corrections and convert recurring problems into new reviewed test cases. Adding those examples creates a new dataset version, so preserve the earlier baseline for future comparisons. Archive the release decision with enough information to reproduce the test under appropriate access controls. The benefit of dataset versioning is not a better-looking dashboard; it is a trustworthy answer to what actually changed between two evaluations.
How We Research
For this November 2026 editorial review, we consulted the named official product documentation: LangChain Docs: Manage datasets in LangSmith; LangChain Docs: How to evaluate agents. We compared the described input and output fields, documented workflow limitations, access and permission controls, and human verification steps to the article scenario. All people, source records, images, schematic states, labels, tool outcomes, and example identifiers are original hypothetical teaching materials, not captured vendor screenshots or measured experimental results. We did not run a live provider API, inspect real museum or volunteer data, establish legal permissions, or measure recognition, translation, classification, retrieval, or evaluation accuracy. Before implementing the workflow, confirm the latest documentation, plan eligibility, pricing, availability, quotas, supported regions and languages, and your organization’s data protection requirements. A qualified human must check source correctness, publication permissions, accessibility, and the implications of any automated operation.
Frequently Asked Questions
Yes. LangSmith documents version creation when examples are added, updated, or deleted.
You can inspect them, but comparisons of model behavior should use matched inputs or clearly separate changes in the test set.
It gives an important version a readable label; retain an exact version reference and a policy for whether tags may move.
No. Source accuracy, rights, provenance, accessibility, and curatorial approval still require human review.
Create a reviewed dataset revision, preserve the old test baseline, and report source changes separately from model regressions.