A museum assistant answers visitor questions about photography. The main visitor rule is easy to retrieve, but an exception for loaned artworks appears in a separate appendix with different wording. OpenAI’s file_search tool has ranking_options that can adjust the ranker and score threshold, as well as max_num_results. Tightening a relevance cutoff may reduce distracting excerpts, but it can also hide the clause that changes the answer. This practical guide treats retrieval configuration as something to evaluate against known evidence, not a magic accuracy dial.
Define the exact question and required evidence
Write a specific visitor question, such as whether flash photography is allowed during a temporary exhibition that includes borrowed objects. Create a source inventory of the general policy, the loan agreement’s permitted public wording and any current exhibition-specific restrictions. Mark one or more passages as mandatory evidence before testing the assistant. This prevents a concise but incomplete answer from being considered successful merely because its primary citation looks plausible.
Include document version, effective dates, public access status and curator-approved wording in your source records. A hypothetical rule may allow photography in permanent galleries while an exception limits photography of one loan. Those details are fictional teaching examples, not claims about any museum’s real policies. If a document is restricted, solve authorization first rather than hoping a ranker makes it less likely to appear.
Understand the supported file_search controls
The OpenAI Responses API reference documents a file_search tool with vector_store_ids, optional max_num_results and ranking_options. The ranking options include a ranker choice such as auto and a score_threshold between zero and one. Higher thresholds attempt to retain only more relevant matches and may return fewer results. The current API also documents hybrid-search weighting controls, depending on the request shape.
A similarity score is not a probability that the passage is true, legally valid or safe to publish. Do not infer from a threshold of 0.7 that a result is 70 percent reliable. Record the exact API shape and supported ranker values for the selected release rather than copying an undocumented default. Costs and output behavior can evolve; benchmark with representative, permitted materials.
Index only approved editions before tuning ranking
Load source files using a controlled naming and versioning scheme. Record which policy edition is current and which appendix is relevant; avoid indexing a mixture of public and internal approvals without appropriate access controls. File search ranking only orders eligible matches and cannot be asked to decide who should be allowed to retrieve a confidential loan agreement. Use trusted file and tenant selection, and separate public statements from private source terms.
When a document is replaced, make sure old content has been removed or marked inactive according to the store’s lifecycle. A high-ranking obsolete paragraph can be more dangerous than a low-ranking current one. Keep a reviewed mapping from returned file IDs to titles, dates and permission states. The result UI should display enough provenance that a reader can locate the exact policy section behind an answer.
Build controlled threshold and result-cap comparisons
Use the same fictional question and fixed source set to compare a baseline ranking, a higher score_threshold and a larger or smaller max_num_results setting. Ask whether the main rule and the important exception both appear among retrieved passages. Measure passage coverage and irrelevant hits separately; a threshold that removes all noise but also removes the exception has failed the editorial purpose.
Avoid reporting a single universal cutoff. An exhibition guide with highly repetitive wording behaves differently from a collection with rare technical terms. Establish a review table: question ID, required source IDs, returned IDs, excluded IDs, observed false matches and final decision. If important evidence is absent, revise the query, indexing or threshold deliberately and re-run the entire fixed test set.
Illustrate the false confidence of overly strict ranking
In the first state, a permissive search returns the rule alongside distracting nearby paragraphs. In the second, an aggressive threshold returns a polished rule but excludes the essential loan exception. The third state retains both the general rule and exception after review and shows source links to each. These states are conceptual, not actual OpenAI ranked outputs or measured performance improvements.
A missing excerpt should never be interpreted as proof that no exception exists. The user-facing answer may need to say that the loan-specific term could not be verified. Design the interface to surface that uncertainty rather than silently issuing permission. Keep citations tied to claims and inspect whether each linked passage supports the language actually presented to the visitor.
Separate relevance ranking from safety and authorization
The search ranker can improve which chunks surface; it does not decide legal permission, approve an exhibition restriction or prevent an unauthorized user from reaching a vector store. Apply source authorization through trusted application controls before the tool receives store IDs and filenames. Ensure the assistant cannot override its public evidence scope by suggesting another store or by inserting adversarial instructions from a retrieved file.
Consider prompt injection inside a policy attachment: a malicious paragraph may ask the assistant to ignore other rules or reveal internal data. Treat retrieved text as reference data only. A safe assistant cites approved passages, rejects unsupported permission requests and routes unresolved cases to staff. Search tuning is an information-retrieval decision, not a substitute for access logs, permission checks or editorial authorization.
Test misses, ambiguity and revision drift
Construct synthetic cases covering exact terminology, synonyms, an appendix with unusual phrasing, an old policy edition, a missing source and a contradictory passage. Label mandatory evidence for each question independently. Run the same cases whenever documents, rankers or query prompts change. Track how often a required exception is excluded, not just whether the response sounds helpful.
Review results in the application’s own environment. A citation may not include the full context needed to interpret an exception, so open surrounding text before release. Document the tool configuration, file IDs and evaluator decisions. The article reports no real OpenAI recall percentages and did not call the API; the diagrams and scenarios show what to test and how to interpret its limitations.
Publish only an answer whose sources cover the condition
Before using a policy answer in a public kiosk, confirm the current general rule, each relevant exception, permission to display the wording and a named reviewer. If the answer could affect photography permissions, quote or paraphrase the exception accurately without presenting the retrieval score as an authorization. Provide a route to museum staff when the indexed evidence is incomplete or contradictory.
Recheck the official Responses API reference for supported ranking_options, result limits and model/tool compatibility immediately before implementation. Keep documented retrieval experiments with source revisions so future staff can understand why a configuration changed. The meaningful success criterion is complete, permitted and traceable evidence, not simply a shorter list of search hits.
How We Research
We researched this November 2026 article from named official documentation: OpenAI: Responses API create reference; OpenAI: File search guide; OpenAI: Retrieval guide. We compared documented request parameters and operational limitations with a hypothetical community organization use case, then designed independent human review and regression checks. Images are original fictional educational diagrams, not product screenshots, actual measured benchmark charts or verified customer cases. We did not submit live API requests, upload personal files, process museum assets or measure vendor accuracy. Implementers must verify current endpoint versions, access controls, region support, pricing and retention before deployment. Factual and rights-sensitive results require review of primary sources.
Frequently Asked Questions
No. It can reduce irrelevant matches while also hiding the passage that qualifies the main rule.
No. It is a ranking signal and does not verify a claim or policy permission.
No. A larger cap may help coverage but cannot substitute for a labeled retrieval evaluation.
No. Enforce authorization and store selection outside the model.
State that the required source was not verified and request human review instead of inferring permission.