To build an AI chatbot, start with one job and one trusted information source. Do not begin by connecting a language model to every internal system. A modest document-answering bot can demonstrate value while keeping its failure modes observable: it should retrieve an approved passage, answer within that passage and abstain when the answer is missing.
This is a system-design guide, not a claim that one code snippet will produce a production-ready agent. It explains the components, tests and release gates to use whether you choose a managed platform or implement your own application with a model API.
Define one job and a measurable success condition
Write an input-output contract before selecting a model: “Given a question about our published return rules, answer from the current policy and cite the correct section; otherwise say no matching policy was found.” Explicitly exclude account lookups, refunds and external actions from the first version. The narrower the scope, the easier it is to evaluate failure.
Create a small corpus of representative questions with reviewer-approved answers and links to the source paragraphs. Include paraphrases, misspellings, conflicting policy versions and questions that should not be answered. These cases are your starting evaluation set, not a promise of coverage for every future user query.
Choose your architecture: scripted, model-only or retrieval
A scripted flow is suitable for a handful of fixed intents. A model-only interface can rephrase instructions but has no dependable access to your private policies. Retrieval-augmented generation, or RAG, lets the application search a controlled collection of documents and supply relevant passages as context before the model responds. It reduces reliance on unsupported recall but does not guarantee truth.
The typical path is: incoming question, identity and policy check, retrieval, passage selection, answer generation, source display, logging and optional escalation. Keep access control before retrieval, not just in a prompt telling the model to be discreet. No model should be the security boundary for a confidential index.
Prepare clean, versioned source material
Gather only documents you are authorized to use. Remove duplicates, expired policies and private fields. Give each item a stable identifier, owner, effective date and permission label. Segment long files so retrieved passages remain meaningful and include the headings needed to interpret exceptions. Indexing a whole PDF without section context can produce misleading excerpts.
When a policy changes, update the indexed version and rerun tests that depend on it. A customer-facing bot should not quote an old refund rule because a search engine ranked it highly. Versioning also enables incident review: you can reconstruct which document was available when a particular answer was generated.
Implement retrieval and citations carefully
Choose an embedding or search service based on access control, document format and operational cost. Retrieve more than one plausible passage if needed, but do not dump an unbounded set of unrelated chunks into every prompt. Ask the model to identify the exact supporting passage, distinguish policy facts from inference and return “not found” when the source is insufficient.
Display citations that a user can actually open where permissions allow. Validate that the citation points to the passage used, rather than a vaguely related document. A retrieved source may itself contain malicious instructions; treat it as data to quote or summarize, not authority to override application rules.
Build an evaluation set before building a dashboard
Include positive cases with unambiguous answers, negative cases with no answer, document conflicts, prompt injection attempts, long questions, multiple languages when relevant and requests for restricted records. Ask a human reviewer to assess grounding, completeness, appropriate abstention and whether access rules were respected. Separate retrieval failures from generation failures.
Use the same cases after every prompt, model or index change. Record version identifiers and reviewer rationale, not merely a green aggregate score. OpenAI’s published evaluation guidance emphasizes task-specific test data and repeated assessment. Five friendly demo questions are not enough for a production reliability claim.
Guardrails belong in code and permissions
Use authentication, role-based retrieval, rate limits, input/output validation and restrictions on external tool calls. For any potentially irreversible operation, validate parameters server-side and require an explicit approval path. Redact unnecessary personal information from logs. Monitor repeated failures and make service owners responsible for incidents.
OWASP documents risks including prompt injection, information disclosure, excessive agency and unbounded resource consumption. A user telling a bot to ignore rules, or an indexed document containing hidden instructions, should not grant new permissions. Prompts can guide behavior but cannot replace technical access controls.
Design the human fallback and failure response
Write a standard response for missing sources: explain the limitation, suggest an official document and offer escalation if the user needs a decision. For errors, avoid displaying internal instructions or raw stack traces. If the retrieval service fails, degrade to a truthful unavailable message instead of quietly making up policy.
If the chatbot handles business requests, preserve enough context for a support agent to continue without asking everything again. Escalation should be a measurable event: a ticket with an owner or a visible contact channel, not an unsupported statement that “someone has been notified”.
Release in stages and keep an exit plan
Run offline evaluations, then an employee-only pilot, then a small low-risk audience. Review incorrect answers and empty-result cases. Establish rollback conditions such as policy hallucinations, unauthorized tool use or rising unresolved contacts. Have a way to disable generation while leaving a static help link available.
Keep exports of approved source documents, test cases and configuration so you can move providers if needed. A chatbot is maintained software: source updates, model changes and new user behavior all create fresh work. Budget for monitoring and content ownership, not only API tokens.
How We Research
We checked retrieval mechanics in OpenAI’s official file-search documentation and drew the evaluation design from OpenAI’s evaluation best practices and the threat categories from OWASP’s Top 10 for LLM applications. We checked Intercom’s handoff configuration documentation as a concrete example of escalation behavior. The architecture, case matrix and rollout conditions are LabFin editorial recommendations, not an installed system, measured latency result or promised security certification.
Frequently Asked Questions
Usually not for a first document-answering chatbot. A model API with controlled retrieval over approved content is often sufficient; fine-tuning and custom training are separate decisions.
No. Retrieval can ground a response in current documents, but a model can still select the wrong passage or misstate an exception. Evaluation and abstention behavior remain necessary.
Test source-supported answers, missing answers, conflicting versions, prompt injection, permission boundaries, handoff and system outages using reviewed cases and a repeatable procedure.
Enforce user authorization before retrieval, label content by permission, minimize logging of personal information and verify that citations cannot reveal documents the requester cannot access.
If you only need a few stable answers or links, a documented FAQ or rules-based flow may be cheaper and safer. Build a model-backed system only after identifying a real need for language flexibility or retrieval.