A chatbot for company documents: prepare the knowledge base and test it

Before building a chatbot for company documents, decide which questions it must answer. “Know every company document” gives you nothing to test against. “Explain the current CRM access procedure and link to the relevant section” does.

RAG retrieves relevant passages from your documents and hands them to a model together with the question. The answer can carry source links, but a link is no proof that the answer is right. The system may have retrieved the wrong document or misread the right one.

• Microsoft — Retrieval augmented generation

Start with a manageable set of current documents

Assign someone to maintain the knowledge base and choose a limited set of current documents. An internal assistant might cover work tools and access requests, while a customer support bot uses service descriptions and approved answers. They do not need to share one collection that everyone can read.

Give each document a title, a version or effective date, an owner, an audience and a stable location. Decide how archives are handled, so that an old instruction cannot quietly replace the current one. Where documents contradict each other, the process owner has to settle it.

Show evidence with the answer

Link to the material that actually supports the statement, ideally the exact section or page. For rules that change, show the version or date so employees can tell current guidance from old.

Decide what should happen when the documents do not contain an answer. The bot could ask for clarification or explain what is missing and direct the user to the responsible person. Agree this before testing so a confident but unsupported answer is not counted as a success.

Apply permissions before the model receives documents

Enforce restrictions at retrieval. An instruction telling the bot not to show restricted documents is not enough. Microsoft documents filtering search results by the groups a user belongs to as an access-control mechanism of its own.

• Microsoft — Document security filters

Test with an ordinary employee account, one with additional permissions and one whose access has been revoked. Check that answers and source links do not reveal restricted documents. Limit access to test logs too, since they may contain private material.

A successful retrieval from our real company service content

On October 7, 2026 we captured our real public service pages. In one selected successful retrieval check, a question about the standalone GEO audit fee returned a passage stating the published $825–$1,650 range with surrounding terms and its source URL. The trace can be inspected in the linked record.

This selected case demonstrates retrieval of a specific passage. It does not measure overall chatbot accuracy or test private permissions. Source support, generated answers and permission handling are separate acceptance questions for the actual project.

• Read the method, results and limitations
• Selected successful retrieval trace · JSON
• Custom RAG scope and handover

In an actual project, prepare acceptance questions from authorised records and hold a separate set back for later testing. Store retrieval results and final answers separately. Permission and freshness requirements need their own evidence from the real deployment environment.

Maintain the knowledge after launch

Assign someone to handle updates and agree how soon new information should appear in answers. Replacing a file does not mean the system has processed it. Check the processing status and any errors, then ask a question whose answer has changed. Test document deletion and permission changes as well.

Investigate where each unsuccessful answer went wrong. If the bot found an old instruction, check document versions and updates. If it found the current instruction but misstated a rule, save the passage and answer for review. If no document answers the question, ask the content owner to fill the gap. Track requests passed to a person too. A confidence score from the model does not tell you which of these problems occurred.

When a separate bot may be unnecessary

For a few short documents that people rarely consult, navigation and ordinary search may be enough. A bot is worth considering when questions repeat, sources multiply and answers have to be pieced together from several sections. Test that need against actual requests.

To discuss a RAG project with Neaptide, bring sample questions with expected answers, the current documents and the list of user groups with their permissions. That is enough to estimate data preparation, integrations, testing and maintenance.

• Discuss a knowledge chatbot

• Understand AI implementation costs

• Check AI answers