A chatbot for company documents: prepare the knowledge base and test it
Neaptide · September 7, 2026 · 6 min read
Current documents, access rights, source links and acceptance tests: how to scope a company knowledge chatbot, with a fictional test matrix.
On this page
Before building a chatbot for company documents, decide which questions it must answer. “Know every company document” gives you no clear way to check completion. “Explain the current CRM access procedure and link to the relevant section” does.
RAG retrieves relevant document passages and supplies them to a model with the question. Answers can include source links, but retrieval and a citation alone do not establish correctness.
Start with a manageable set of current documents
Assign someone to maintain the knowledge base and choose a limited set of current documents. An internal assistant might cover work tools and access requests, while a customer support bot uses service descriptions and approved answers. They do not need to share one collection that everyone can read.
Give documents titles, versions or effective dates, owners, audiences and stable locations. Decide how archives are handled. Historical instructions should not silently replace current ones. A process owner must resolve contradictions.
Show evidence with the answer
Link to the material that actually supports the statement, ideally the section or page. Show version or date for changing rules so employees can check differences from old guidance.
Decide what should happen when the documents do not contain an answer. The bot could ask for clarification or explain what is missing and direct the user to the responsible person. Agree this before testing so a confident but unsupported answer is not counted as a success.
Apply permissions before the model receives documents
Enforce restrictions during retrieval, not merely through an instruction to the bot. Microsoft documents filtering results by permitted user groups as part of access control.
Test with an ordinary employee account, one with additional permissions and one whose access has been revoked. Check that answers and source links do not reveal restricted documents. Limit access to test logs too, since they may contain private material.
Five fictional acceptance cases
These invented rules are not Neaptide policy. Version 1 requested CRM access by email; current version 2 uses a service portal and manager approval. A separate administrator instruction is restricted.
| Question or condition | Expected behaviour | Check |
|---|---|---|
| How do I get CRM access? | Service portal and manager approval | Version 2 and the supporting section |
| Can I use the old email? | Explain the current process | Do not present the archive as current |
| No delivery time is documented | Do not invent a timeframe | Clarify or refer to the owner |
| Employee asks about a restricted setting | Do not disclose restricted information | No forbidden passage or bypass link |
| Version 3 replaces version 2 | Use the new guidance after the agreed refresh | Processing status and retest; no stale answer treated as current |
Prepare cases before tuning and retain some for final testing. Rehearsing only a few convenient questions tells you little about new requests. Keep retrieval results and final answers separately to locate errors.
Maintain the knowledge after launch
Assign someone to handle updates and agree how soon new information should appear in answers. Replacing a file does not mean the system has processed it. Check the processing status and any errors, then ask a question whose answer has changed. Test document deletion and permission changes as well.
Review questions the bot could not answer from the documents, incorrect search results and requests passed to a person. The source instructions may need fixing rather than the model. A confidence score supplied by the model is not evidence that its answer is correct.
When a separate bot may be unnecessary
For a few short documents used infrequently, navigation and ordinary search may suffice. Consider a bot when questions repeat, sources multiply or answers span sections. Test that need against actual requests.
For a Neaptide RAG discussion, bring questions, current documents, user groups and expected answers. This supports estimates for data preparation, integrations, testing and maintenance.