OpenAI Vector Store score_threshold: Do Not Filter Out a Quiet Conservation Warning

A cleaner result list can be an unsafe result list

A museum conservator asks an internal assistant whether a solvent is appropriate for a painted display mount. The current manual contains a prominent treatment procedure and a short warning in an appendix explaining when not to use the solvent. A search result page full of highly ranked procedures might look reassuring, yet the necessary exception could be scored lower because its wording differs from the question. OpenAI vector store search allows a ranking_options.score_threshold setting to filter matches by their relevance scores. This is a retrieval control, not a safety certification. The practical goal is to preserve access to the appendix warning, identify the correct manual revision and refuse a confident answer when the relevant exception cannot be found.

Read the score threshold as a cutoff, not confidence

The OpenAI vector store search endpoint accepts ranking_options with a score_threshold in the range from zero to one, alongside an optional ranker. Raising the threshold restricts the results and can exclude potentially useful chunks. The response can include a file ID, filename, attributes, content and a score for each returned item. Those scores describe retrieval ranking; they are not calibrated probabilities that a treatment is safe. Check the specific API surface before copying settings from the Responses file_search tool, which has related but separately documented options. Do not state that a particular numeric threshold is universally safe: acceptable recall depends on the collection and on the consequences of a missed warning.

Create a controlled set of manual versions

Before tuning the cutoff, assemble a modest fixture set from approved, rights-cleared documents. Include an active procedure, its cautionary appendix, a superseded version and a neighboring object’s treatment sheet that uses similar vocabulary. Attach document attributes for accession, approval state and revision where those metadata are reliable. Keep all candidates in a staging vector store and wait until indexing is complete. Document who decides which warnings must be retrieved for each question. This is an editorial relevance judgment, not a value the ranking model discovers independently. A reference spreadsheet should name the expected source file IDs and passages so results can be reviewed without memorizing a model’s output.

Run paired searches without changing the corpus

For each question, send the same query and file-attribute restrictions with two proposed score thresholds, such as 0.0 and 0.5, then inspect the response items. These numbers are illustrative settings, not measured optimal thresholds. Preserve the ranking model option, dataset revision and max_num_results so that only the threshold changes. Query variants should cover the formal chemical name, a technician’s nickname and a question written as a prohibited action. Log which approved passages remain visible, which disappear and which unrelated versions appear. If a warning vanishes at the stricter cutoff, diagnose that omission before considering the cleaner list an improvement. No live benchmark was performed to support hypothetical counts.

Compare three states instead of declaring a winner

The accompanying comparison illustrates a broad shortlist containing both a procedure and its warning, a stricter shortlist missing that warning and a corrected editorial review that retrieves the exact appendix. The middle state depicts a plausible failure mode, not an assertion that OpenAI always drops warnings at one specified threshold. A high-scoring passage can still belong to the wrong object. An adequate final state therefore requires evidence of the correct accession, policy revision and readable caution text. When reviewers disagree on whether the appendix is required for a prompt, resolve that disagreement in the fixture documentation rather than silently changing the threshold until the answer appears comfortable.

Separate relevance controls from permission controls

Attribute filters and server-side authorization decide which records a user may inspect; the score threshold only narrows the ranked matches among eligible sources. Never use a high threshold to conceal restricted information, and never assume a filtered-out private record has become inaccessible. For a visitor-facing chatbot, do not place confidential treatment notes in a public retrieval scope at all. For internal conservators, include the correct approved material and let the application enforce role access. Keep the user’s question and the search parameters in an access-controlled audit log, while avoiding unrestricted copies of donor or treatment records. One setting cannot replace permissions, source approval and review of the generated answer.

Design hard negative cases and no-match behavior

Include questions that should return no authorized instruction, such as a chemical not mentioned in the approved manual or an accession number from a different collection. Also test a query that has a close semantic match but conflicts with a caution. A strong result for a neighbor is not permission to infer the missing rule. Record whether the search returns zero matches, a short but incomplete list or a misleadingly fluent passage. Require the assistant to surface uncertainty and route the request to collection staff when evidence is missing. This rule should be evaluated separately from whether a low-score result crossed the retrieval threshold. An empty list is often a review event, not a successful safety decision.

Recheck after edits to chunks, metadata or ranking

Any change in chunking strategy, source-file revisions, metadata filters or ranking behavior can change the shortlist even when the threshold number stays fixed. Version the fixture questions and expected file IDs alongside the service configuration. During a release review, compare the current result set with the last accepted set, especially for cases that previously depended on a lower-ranked warning. Inspect the actual passages rather than accepting a matching filename as sufficient. If a policy revision changes the caution text, update the gold reference from the approved document before rerunning tests. Keep performance and storage cost measurements separate; no synthetic diagram should be treated as an empirical latency study.

Release on evidence coverage, not tidy rankings

The acceptance record should contain the tested threshold, corpus revision, approved document identifier, warning passage and reviewer decision for every critical question. Permit a stricter cutoff only when the required caution remains retrievable and a no-match outcome cannot be turned into an unsupported recommendation. For lower-risk catalog searches, a different threshold may be reasonable, but do not reuse that configuration for chemical treatment instructions by convenience. If the shortlist fails an exception case, keep the previous configuration or widen retrieval pending investigation. The final answer must cite the current manual and explicitly acknowledge limits. In conservation workflows, the missing appendix matters more than a visually shorter list of search hits.

Related guides

• OpenAI Vector Store Expiration: Retire a Temporary Grant Index Without Losing Source Records
• OpenAI Vector Store Chunking: Keep Safety Warnings With Museum Repair Steps
• OpenAI Vector Store File Delete: Withdraw an Exhibit Note Without Deleting Its Original

How We Research

For this guide we consulted the official documentation: OpenAI vector store search API; OpenAI Retrieval guide. We compared named fields, supported behaviors and stated limitations, then developed an instructional museum scenario with explicit failure states. We did not run a live vendor API benchmark, inspect a production museum system, or measure vendor performance. The workflow and illustrations are proposed review steps; verify API-version support and your organization’s current policy before applying them.

Frequently Asked Questions